| Spouse | Sarah Elizabeth "Sally" Carter (1776 - 1838) |
| Child | George Grayson (1794 - 1856) |
| Child | Mary Polly Grayson (1795 - 1846) |
| Child | James John Grayson (1797 - 1797) |
| Child | William Grayson (1797 - 1806) |
| Child | John Cullen Grayson (1799 - 1849) |
| Child | Henry Grayson (1800 - ) |
| Child | Robert Grayson (1801 - 1810) |
| Child | Charles C. Grayson (1803 - 1856) |
| Child | Jackson L. Grayson (1805 - 1853) |
| Child | Nancy Grayson (1808 - 1867) |
| Child | Ambrose G. Grayson (1810 - 1853) |
| Child | Sarah Ann Grayson (1811 - 1860) |
| Child | Napoleon Bonaparte Grayson (1815 - 1858) |
| Child | James Gordon Grayson (1817 - 1864) |
| Child | Benjamin Grayson (1822 - 1825) |