Working with Economic data in Python

This notebook will introduce you to working with data in Python. You will use packages like Numpy to manipulate, work and do computations with arrays, matrices, and such, and anipulate data (see my Introduction to Python). But given the needs of economists (and other scientists) it will be advantageous for us to use pandas. pandas is an open source, BSD-licensed library providing high-performance, easy-to-use data structures and data analysis tools for Python. pandas allows you to import and process data in many useful ways. It interacts greatly with other packages that complement it making it a very powerful tool for data analysis.

With pandas you can

1. Import many types of data, including
   • CSV files
   • Tab or other types of delimited files
   • Excel (xls, xlsx) files
   • Stata files

2. Open files directly from a website
3. Merge, select, join data
4. Perform statistical analyses
5. Create plots of your data

and much more. Let's start by importing pandas and use to it download some data and create some of the figures from the lecture notes. Note that when importing pandas it is accustomed to assign it the alias pd. I suggest you follow this conventiuon, which will make using other peoples code and snippets easier.

# Let's import pandas and some other basic packages we will use 
from __future__ import division
%pylab --no-import-all
%matplotlib inline
import pandas as pd
import numpy as np

Using matplotlib backend: <object object at 0x147549690>
%pylab is deprecated, use %matplotlib inline and import the required libraries.
Populating the interactive namespace from numpy and matplotlib

Working with Pandas

The basic structures in pandas are pd.Series and pd.DataFrame. You can think of a pd.Series as a labeled vector that contains data and has a large set of functions that can be easily performed on it. A pd.DataFrame is similar a table/matrix of multidimensional data where each column contains a pd.Series. I know...this may not explain much, so let's start with some actual examples. Let's create two series, one containing some country names and another containing some ficticious data.

countries = pd.Series(['Colombia', 'Turkey', 'USA', 'Germany', 'Chile'], name='country')
print(countries)
print('\n', 'There are ', countries.shape[0], 'countries in this series.')

0    Colombia
1      Turkey
2         USA
3     Germany
4       Chile
Name: country, dtype: object

 There are  5 countries in this series.

Notice that we have assinged a name to the series that is different than the name of the variable containing the series. Our print(countries) statement is showing the series and its contents, its name and the dype of data it contains. Here our series is only composed of strings so it assigns it the object dtype (not important for now, but we will use this later to convert data between types, e.g. strings to integers or floats or the other way around).

Let's create the data using some of the functions we already learned.

np.random.seed(123456)
data = pd.Series(np.random.normal(size=(countries.shape)), name='noise')
print(data)
print('\n', 'The average in this sample is ', data.mean())

0    0.469112
1   -0.282863
2   -1.509059
3   -1.135632
4    1.212112
Name: noise, dtype: float64

 The average in this sample is  -0.24926597871826645

Here we have used the mean() function of the series to compute its mean. There are many other properties/functions for these series including std(), shape, count(), max(), min(), etc. You can access these by writing series.name_of_function_or_property. To see what functions are available you can hit tab after writing series..

Let's create a pd.DataFrame using these two series.

df = pd.DataFrame([countries, data])
df

	0	1	2	3	4
country	Colombia	Turkey	USA	Germany	Chile
noise	0.469112	-0.282863	-1.509059	-1.135632	1.212112

Not exactly what we'd like, but don't worry, we can just transpose it so it has each country with its data in a row.

df = df.T
df

	country	noise
0	Colombia	0.469112
1	Turkey	-0.282863
2	USA	-1.509059
3	Germany	-1.135632
4	Chile	1.212112

Now let us add some more data to this dataframe. This is done easily by defining a new columns. Let's create the square of noise, create the sum of noise and its square, and get the length of the country's name.

df['noise_sq'] = df.noise**2
df['noise and its square'] = df.noise + df.noise_sq
df['name length'] = df.country.apply(len)
df

	country	noise	noise_sq	noise and its square	name length
0	Colombia	0.469112	0.220066	0.689179	8
1	Turkey	-0.282863	0.080012	-0.202852	6
2	USA	-1.509059	2.277258	0.768199	3
3	Germany	-1.135632	1.289661	0.154029	7
4	Chile	1.212112	1.469216	2.681328	5

This shows some of the ways in which you can create new data. Especially useful is the apply method, which applies a function to the series. You can also apply a function to the whole dataframe, which is useful if you want to perform computations using various columns.

Let's see some other ways in which we can interact with dataframes. First, let's select some observations, e.g., all countries in the South America.

# Let's create a list of South American countries
south_america = ['Colombia', 'Chile']
# Select the rows for South American countries
df.loc[df.country.apply(lambda x: x in south_america)]

	country	noise	noise_sq	noise and its square	name length
0	Colombia	0.469112	0.220066	0.689179	8
4	Chile	1.212112	1.469216	2.681328	5

Now let's use this to create a dummy indicating whether a country belongs to South America. To understand what is going on let's show the result of the condition for selecting rows.

df.country.apply(lambda x: x in south_america)

0     True
1    False
2    False
3    False
4     True
Name: country, dtype: bool

So in the previous selection of rows we told pandas which rows we wanted or not to be included by passing a series of booleans (True, False). We can use this result to create the dummy, we only need to convert the output to int.

df['South America'] = df.country.apply(lambda x: x in south_america).astype(int)

Now, let's plot the various series in the dataframe

df.plot()

<Axes: >

Not too nice nor useful. Notice that it assigned the row number to the x-axis labels. Let's change the row labels, which are contained in the dataframe's index by assigning the country names as the index.

df = df.set_index('country')
print(df)
df.plot()

             noise  noise_sq noise and its square  name length  South America
country                                                                      
Colombia  0.469112  0.220066             0.689179            8              1
Turkey   -0.282863  0.080012            -0.202852            6              0
USA      -1.509059  2.277258             0.768199            3              0
Germany  -1.135632  1.289661             0.154029            7              0
Chile     1.212112  1.469216             2.681328            5              1

<Axes: xlabel='country'>

Better, but still not very informative. Below we will improve on this when we work with some real data.

Notice that by using the set_index function we have assigned the index to the country names. This may be useful to select data. E.g., if we want to see only the row for Colombia we can

df.loc['Colombia']

noise                   0.469112
noise_sq                0.220066
noise and its square    0.689179
name length                    8
South America                  1
Name: Colombia, dtype: object

Getting data

One of the nice features of pandas and its ecology is that it makes obtaining data very easy. In order to exemplify this and also to revisit some of the basic facts of comparative development, let's download some data from various sources. This may require you to create accounts in order to access and download the data (sometimes the process is very simple and does not require an actual project...in other cases you need to propose a project and be approved...usually due to privacy concerns with micro-data). Don't be afraid, all these sources are free and are used a lot in research, so it is good that you learn to use them. Let's start with a list of useful sources.

Country-level data economic data

• World Bank provides all kinds of socio-economic data.
• Penn World Tables is a database with information on relative levels of income, output, input and productivity, covering 182 countries between 1950 and 2017.
• Maddison Historical Data provides the most used historical statistics on population and GDP
• The Maddison Project Database provides information on comparative economic growth and income levels over the very long run, follow-up to Maddison.
• Comparative Historical National Accounts provides information on Gross Domestic Product, including an industry breakdown, for the 19th and 20th centuries.
• Human Mortality Database provides detailed mortality and population data for the world for the last two centuries.

Censuses, Surveys, and other micro-level data

• IPUMS: provides census and survey data from around the world integrated across time and space.
• General Social Survey provides survey data on what Americans think and feel about such issues as national spending priorities, crime and punishment, intergroup relations, and confidence in institutions.
• European Social Survey provides survey measures on the attitudes, beliefs and behaviour patterns of diverse European populations in more than thirty nations.
• UK Data Service is the UK’s largest collection of social, economic and population data resources.
• SHRUG is The Socioeconomic High-resolution Rural-Urban Geographic Platform for India. Provides access to dozens of datasets covering India’s 500,000 villages and 8000 towns using a set of a common geographic identifiers that span 25 years.

Divergence - Big time

To study the divergence across countries let's download and plot the historical GDP and population data. In order to keep the data and not having to download it everytime from scratch, we'll create a folder ./data in the currect directory and save each file there. Also, we'll make sure that if the data does not exist, we download it. We'll use the os package to create directories.

Setting up paths

import os

pathout = './data/'

if not os.path.exists(pathout):
    os.mkdir(pathout)
    
pathgraphs = './graphs/'
if not os.path.exists(pathgraphs):
    os.mkdir(pathgraphs)

Download New Maddison Project Data

try:
    maddison_new = pd.read_stata(pathout + 'Maddison2020.dta')
    maddison_new_region = pd.read_stata(pathout + 'Maddison2018_region.dta')
    maddison_new_1990 = pd.read_stata(pathout + 'Maddison2018_1990.dta')
except:
    maddison_new = pd.read_stata('https://www.rug.nl/ggdc/historicaldevelopment/maddison/data/mpd2020.dta')
    maddison_new.to_stata(pathout + 'Maddison2020.dta', write_index=False, version=117)
    maddison_new_region = pd.read_stata('https://www.rug.nl/ggdc/historicaldevelopment/maddison/data/mpd2018_region_data.dta')
    maddison_new_region.to_stata(pathout + 'Maddison2018_region.dta', write_index=False, version=117)
    maddison_new_1990 = pd.read_stata('https://www.rug.nl/ggdc/historicaldevelopment/maddison/data/mpd2018_1990bm.dta')
    maddison_new_1990.to_stata(pathout + 'Maddison2018_1990.dta', write_index=False, version=117)

maddison_new

	countrycode	country	year	gdppc	pop
0	AFG	Afghanistan	1820	NaN	3280.00000
1	AFG	Afghanistan	1870	NaN	4207.00000
2	AFG	Afghanistan	1913	NaN	5730.00000
3	AFG	Afghanistan	1950	1156.0000	8150.00000
4	AFG	Afghanistan	1951	1170.0000	8284.00000
...	...	...	...	...	...
21677	ZWE	Zimbabwe	2014	1594.0000	13313.99205
21678	ZWE	Zimbabwe	2015	1560.0000	13479.13812
21679	ZWE	Zimbabwe	2016	1534.0000	13664.79457
21680	ZWE	Zimbabwe	2017	1582.3662	13870.26413
21681	ZWE	Zimbabwe	2018	1611.4052	14096.61179

21682 rows × 5 columns

This dataset is in long format. Also, notice that the year is not an integer. Let's correct this

maddison_new['year'] = maddison_new.year.astype(int)
maddison_new

	countrycode	country	year	gdppc	pop
0	AFG	Afghanistan	1820	NaN	3280.00000
1	AFG	Afghanistan	1870	NaN	4207.00000
2	AFG	Afghanistan	1913	NaN	5730.00000
3	AFG	Afghanistan	1950	1156.0000	8150.00000
4	AFG	Afghanistan	1951	1170.0000	8284.00000
...	...	...	...	...	...
21677	ZWE	Zimbabwe	2014	1594.0000	13313.99205
21678	ZWE	Zimbabwe	2015	1560.0000	13479.13812
21679	ZWE	Zimbabwe	2016	1534.0000	13664.79457
21680	ZWE	Zimbabwe	2017	1582.3662	13870.26413
21681	ZWE	Zimbabwe	2018	1611.4052	14096.61179

21682 rows × 5 columns

Original Maddison Data

Now, let's download, save and read the original Maddison database. Since the original file is an excel file with different data on each sheet, it will require us to use a different method to get all the data.

if not os.path.exists(pathout + 'Maddison_original.xlsx'):
    import urllib
    dataurl = "https://www.rug.nl/ggdc/historicaldevelopment/maddison/data/md2010_horizontal.xlsx"
    urllib.request.urlretrieve(dataurl, pathout + 'Maddison_original.xlsx')

Some data munging

This dataset is not very nicely structured for importing, as you can see if you open it in Excel. I suggest you do so, so that you can better see what is going on. Notice that the first two rows really have no data. Also, every second column is empty. Moreover, there are a few empty rows. Let's import the data and clean it so we can plot and analyse it better.

maddison_old_pop = pd.read_excel(pathout + 'Maddison_original.xlsx', sheet_name="Population", skiprows=2)
maddison_old_pop

	Unnamed: 0	1	Unnamed: 2	1000	Unnamed: 4	1500	Unnamed: 6	1600	Unnamed: 8	1700	...	2002	2003	2004	2005	2006	2007	2008	2009	Unnamed: 201	2030
0	Western Europe	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
1	Austria	500.0	NaN	700.0	NaN	2000.0	NaN	2500.0	NaN	2500.0	...	8148.312	8162.656	8174.762	8184.691	8192.880	8199.783	8205.533	8210	NaN	8120.000
2	Belgium	300.0	NaN	400.0	NaN	1400.0	NaN	1600.0	NaN	2000.0	...	10311.970	10330.824	10348.276	10364.388	10379.067	10392.226	10403.951	10414	NaN	10409.000
3	Denmark	180.0	NaN	360.0	NaN	600.0	NaN	650.0	NaN	700.0	...	5374.693	5394.138	5413.392	5432.335	5450.661	5468.120	5484.723	5501	NaN	5730.488
4	Finland	20.0	NaN	40.0	NaN	300.0	NaN	400.0	NaN	400.0	...	5193.039	5204.405	5214.512	5223.442	5231.372	5238.460	5244.749	5250	NaN	5201.445
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
273	Guadeloupe	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	...	435.739	440.189	444.515	448.713	452.776	456.698	460.486	n.a.	NaN	523.493
274	Guyana (Fr.)	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	...	182.333	186.917	191.309	195.506	199.509	203.321	206.941	n.a.	NaN	272.781
275	Martinique	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	...	422.277	425.966	429.510	432.900	436.131	439.202	442.119	n.a.	NaN	486.714
276	Reunion	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	...	743.981	755.171	766.153	776.948	787.584	798.094	808.506	n.a.	NaN	1025.217
277	Total	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	...	1784.330	1808.243	1831.487	1854.067	1876.000	1897.315	1918.052	n.a.	NaN	2308.205

278 rows × 203 columns

maddison_old_gdppc = pd.read_excel(pathout + 'Maddison_original.xlsx', sheet_name="PerCapita GDP", skiprows=2)
maddison_old_gdppc

	Unnamed: 0	1	Unnamed: 2	1000	Unnamed: 4	1500	Unnamed: 6	1600	Unnamed: 8	1700	...	1999	2000	2001	2002	2003	2004	2005	2006	2007	2008
0	Western Europe	NaN	NaN	NaN	NaN		NaN	NaN	NaN	NaN	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
1	Austria	425.000000	NaN	425.000000	NaN	707	NaN	837.200000	NaN	993.200000	...	20065.093878	20691.415561	20812.893753	20955.874051	21165.047259	21626.929322	22140.725899	22892.682427	23674.041130	24130.547035
2	Belgium	450.000000	NaN	425.000000	NaN	875	NaN	975.625000	NaN	1144.000000	...	19964.428266	20656.458570	20761.238278	21032.935511	21205.859281	21801.602508	22246.561977	22881.632810	23446.949672	23654.763464
3	Denmark	400.000000	NaN	400.000000	NaN	738.333333	NaN	875.384615	NaN	1038.571429	...	22254.890572	22975.162513	23059.374968	23082.620719	23088.582457	23492.664119	23972.564284	24680.492880	24995.245167	24620.568805
4	Finland	400.000000	NaN	400.000000	NaN	453.333333	NaN	537.500000	NaN	637.500000	...	18855.985066	19770.363126	20245.896529	20521.702225	20845.802738	21574.406196	22140.573208	23190.283543	24131.519569	24343.586318
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
190	Total Africa	472.352941	NaN	424.767802	NaN	413.709504	NaN	422.071584	NaN	420.628684	...	1430.752576	1447.071701	1471.156532	1482.629352	1517.935644	1558.099461	1603.686517	1663.531318	1724.226776	1780.265474
191	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
192	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
193	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
194	World Average	466.752281	NaN	453.402162	NaN	566.389464	NaN	595.783856	NaN	614.853602	...	5833.255492	6037.675887	6131.705471	6261.734267	6469.119575	6738.281333	6960.031035	7238.383483	7467.648232	7613.922924

195 rows × 200 columns

Let's start by renaming the first column, which has the region/country names

maddison_old_pop.rename(columns={'Unnamed: 0':'Country'}, inplace=True)
maddison_old_gdppc.rename(columns={'Unnamed: 0':'Country'}, inplace=True)

Now let's drop all the columns that do not have data

maddison_old_pop = maddison_old_pop[[col for col in maddison_old_pop.columns if str(col).startswith('Unnamed')==False]]
maddison_old_gdppc = maddison_old_gdppc[[col for col in maddison_old_gdppc.columns if str(col).startswith('Unnamed')==False]]

Now, let's change the name of the columns so they reflect the underlying variable

maddison_old_pop.columns = ['Country'] + ['pop_'+str(col) for col in maddison_old_pop.columns[1:]]
maddison_old_gdppc.columns = ['Country'] + ['gdppc_'+str(col) for col in maddison_old_gdppc.columns[1:]]

maddison_old_pop

	Country	pop_1	pop_1000	pop_1500	pop_1600	pop_1700	pop_1820	pop_1821	pop_1822	pop_1823	...	pop_2001	pop_2002	pop_2003	pop_2004	pop_2005	pop_2006	pop_2007	pop_2008	pop_2009	pop_2030
0	Western Europe	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
1	Austria	500.0	700.0	2000.0	2500.0	2500.0	3369.0	3386.0	3402.0	3419.0	...	8131.690	8148.312	8162.656	8174.762	8184.691	8192.880	8199.783	8205.533	8210	8120.000
2	Belgium	300.0	400.0	1400.0	1600.0	2000.0	3434.0	3464.0	3495.0	3526.0	...	10291.679	10311.970	10330.824	10348.276	10364.388	10379.067	10392.226	10403.951	10414	10409.000
3	Denmark	180.0	360.0	600.0	650.0	700.0	1155.0	1167.0	1179.0	1196.0	...	5355.826	5374.693	5394.138	5413.392	5432.335	5450.661	5468.120	5484.723	5501	5730.488
4	Finland	20.0	40.0	300.0	400.0	400.0	1169.0	1186.0	1202.0	1219.0	...	5180.309	5193.039	5204.405	5214.512	5223.442	5231.372	5238.460	5244.749	5250	5201.445
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
273	Guadeloupe	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	...	431.170	435.739	440.189	444.515	448.713	452.776	456.698	460.486	n.a.	523.493
274	Guyana (Fr.)	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	...	177.562	182.333	186.917	191.309	195.506	199.509	203.321	206.941	n.a.	272.781
275	Martinique	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	...	418.454	422.277	425.966	429.510	432.900	436.131	439.202	442.119	n.a.	486.714
276	Reunion	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	...	732.570	743.981	755.171	766.153	776.948	787.584	798.094	808.506	n.a.	1025.217
277	Total	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	...	1759.756	1784.330	1808.243	1831.487	1854.067	1876.000	1897.315	1918.052	n.a.	2308.205

278 rows × 197 columns

maddison_old_gdppc

	Country	gdppc_1	gdppc_1000	gdppc_1500	gdppc_1600	gdppc_1700	gdppc_1820	gdppc_1821	gdppc_1822	gdppc_1823	...	gdppc_1999	gdppc_2000	gdppc_2001	gdppc_2002	gdppc_2003	gdppc_2004	gdppc_2005	gdppc_2006	gdppc_2007	gdppc_2008
0	Western Europe	NaN	NaN		NaN	NaN	NaN	NaN	NaN	NaN	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
1	Austria	425.000000	425.000000	707	837.200000	993.200000	1218.165628	NaN	NaN	NaN	...	20065.093878	20691.415561	20812.893753	20955.874051	21165.047259	21626.929322	22140.725899	22892.682427	23674.041130	24130.547035
2	Belgium	450.000000	425.000000	875	975.625000	1144.000000	1318.870122	NaN	NaN	NaN	...	19964.428266	20656.458570	20761.238278	21032.935511	21205.859281	21801.602508	22246.561977	22881.632810	23446.949672	23654.763464
3	Denmark	400.000000	400.000000	738.333333	875.384615	1038.571429	1273.593074	1320.479863	1326.547922	1307.692308	...	22254.890572	22975.162513	23059.374968	23082.620719	23088.582457	23492.664119	23972.564284	24680.492880	24995.245167	24620.568805
4	Finland	400.000000	400.000000	453.333333	537.500000	637.500000	781.009410	NaN	NaN	NaN	...	18855.985066	19770.363126	20245.896529	20521.702225	20845.802738	21574.406196	22140.573208	23190.283543	24131.519569	24343.586318
...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...	...
190	Total Africa	472.352941	424.767802	413.709504	422.071584	420.628684	419.755914	NaN	NaN	NaN	...	1430.752576	1447.071701	1471.156532	1482.629352	1517.935644	1558.099461	1603.686517	1663.531318	1724.226776	1780.265474
191	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
192	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
193	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	...	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN	NaN
194	World Average	466.752281	453.402162	566.389464	595.783856	614.853602	665.735330	NaN	NaN	NaN	...	5833.255492	6037.675887	6131.705471	6261.734267	6469.119575	6738.281333	6960.031035	7238.383483	7467.648232	7613.922924

195 rows × 195 columns

Let's choose the rows that hold the aggregates by region for the main regions of the world.

gdppc = maddison_old_gdppc.loc[maddison_old_gdppc.Country.apply(lambda x: str(x).upper().find('TOTAL')!=-1)].reset_index(drop=True)
gdppc = gdppc.dropna(subset=['gdppc_1'])
gdppc = gdppc.loc[2:]
gdppc['Country'] = gdppc.Country.str.replace('Total', '').str.replace('Countries', '').str.replace('\d+', '').str.replace('European', 'Europe').str.strip()
gdppc = gdppc.loc[gdppc.Country.apply(lambda x: x.find('USSR')==-1 and  x.find('West Asian')==-1)].reset_index(drop=True)
gdppc

	Country	gdppc_1	gdppc_1000	gdppc_1500	gdppc_1600	gdppc_1700	gdppc_1820	gdppc_1821	gdppc_1822	gdppc_1823	...	gdppc_1999	gdppc_2000	gdppc_2001	gdppc_2002	gdppc_2003	gdppc_2004	gdppc_2005	gdppc_2006	gdppc_2007	gdppc_2008
0	30 Western Europe	576.167665	427.425665	771.093805	887.906964	993.456911	1194.184683	NaN	NaN	NaN	...	18497.208533	19176.001655	19463.863297	19627.707522	19801.145425	20199.220700	20522.238008	21087.304789	21589.011346	21671.774225
1	Western Offshoots	400.000000	400.000000	400	400.000000	476.000000	1201.993477	NaN	NaN	NaN	...	26680.580823	27393.808035	27387.312035	27648.644070	28090.274362	28807.845958	29415.399334	29922.741918	30344.425293	30151.805880
2	7 East Europe	411.789474	400.000000	496	548.023599	606.010638	683.160984	NaN	NaN	NaN	...	5734.162109	5970.165085	6143.112873	6321.395376	6573.365882	6942.136596	7261.721015	7730.097570	8192.881904	8568.967581
3	Latin America	400.000000	400.000000	416.457143	437.558140	526.639004	691.060678	NaN	NaN	NaN	...	5765.585093	5889.237351	5846.295193	5746.609672	5785.841237	6063.068969	6265.525702	6530.533583	6783.869986	6973.134656
4	Asia	455.671021	469.961665	568.4179	573.550859	571.605276	580.626115	NaN	NaN	NaN	...	3623.902724	3797.608955	3927.186275	4121.275511	4388.982705	4661.517477	4900.563281	5187.253152	5408.383588	5611.198564
5	Africa	472.352941	424.767802	413.709504	422.071584	420.628684	419.755914	NaN	NaN	NaN	...	1430.752576	1447.071701	1471.156532	1482.629352	1517.935644	1558.099461	1603.686517	1663.531318	1724.226776	1780.265474

6 rows × 195 columns

Let's drop missing values

gdppc = gdppc.dropna(axis=1, how='any')
gdppc

	Country	gdppc_1	gdppc_1000	gdppc_1500	gdppc_1600	gdppc_1700	gdppc_1820	gdppc_1870	gdppc_1900	gdppc_1913	...	gdppc_1999	gdppc_2000	gdppc_2001	gdppc_2002	gdppc_2003	gdppc_2004	gdppc_2005	gdppc_2006	gdppc_2007	gdppc_2008
0	30 Western Europe	576.167665	427.425665	771.093805	887.906964	993.456911	1194.184683	1953.068150	2884.661525	3456.576178	...	18497.208533	19176.001655	19463.863297	19627.707522	19801.145425	20199.220700	20522.238008	21087.304789	21589.011346	21671.774225
1	Western Offshoots	400.000000	400.000000	400	400.000000	476.000000	1201.993477	2419.152411	4014.870040	5232.816582	...	26680.580823	27393.808035	27387.312035	27648.644070	28090.274362	28807.845958	29415.399334	29922.741918	30344.425293	30151.805880
2	7 East Europe	411.789474	400.000000	496	548.023599	606.010638	683.160984	936.628265	1437.944586	1694.879668	...	5734.162109	5970.165085	6143.112873	6321.395376	6573.365882	6942.136596	7261.721015	7730.097570	8192.881904	8568.967581
3	Latin America	400.000000	400.000000	416.457143	437.558140	526.639004	691.060678	676.005331	1113.071149	1494.431922	...	5765.585093	5889.237351	5846.295193	5746.609672	5785.841237	6063.068969	6265.525702	6530.533583	6783.869986	6973.134656
4	Asia	455.671021	469.961665	568.4179	573.550859	571.605276	580.626115	553.459947	637.615593	695.131881	...	3623.902724	3797.608955	3927.186275	4121.275511	4388.982705	4661.517477	4900.563281	5187.253152	5408.383588	5611.198564
5	Africa	472.352941	424.767802	413.709504	422.071584	420.628684	419.755914	500.011054	601.236364	637.433138	...	1430.752576	1447.071701	1471.156532	1482.629352	1517.935644	1558.099461	1603.686517	1663.531318	1724.226776	1780.265474

6 rows × 70 columns

Let's convert from wide to long format

gdppc = pd.wide_to_long(gdppc, ['gdppc_'], i='Country', j='year').reset_index()
gdppc

	Country	year	gdppc_
0	30 Western Europe	1	576.167665
1	Western Offshoots	1	400.0
2	7 East Europe	1	411.789474
3	Latin America	1	400.0
4	Asia	1	455.671021
...	...	...	...
409	Western Offshoots	2008	30151.80588
410	7 East Europe	2008	8568.967581
411	Latin America	2008	6973.134656
412	Asia	2008	5611.198564
413	Africa	2008	1780.265474

414 rows × 3 columns

Plotting

We can now plot the data. Let's try two different ways. The first uses the plot function from pandas. The second uses the package seaborn, which improves on the capabilities of matplotlib. The main difference is how the data needs to be organized. Of course, these are not the only ways to plot and we can try others.

import matplotlib as mpl
import seaborn as sns
# Setup seaborn
sns.set()

Let's pivot the table so that each region is a column and each row is a year. This will allow us to plot using the plot function of the pandas DataFrame.

gdppc2 = gdppc.pivot_table(index='year',columns='Country',values='gdppc_',aggfunc='sum')
gdppc2

Country	30 Western Europe	7 East Europe	Africa	Asia	Latin America	Western Offshoots
year
1	576.167665	411.789474	472.352941	455.671021	400.0	400.0
1000	427.425665	400.0	424.767802	469.961665	400.0	400.0
1500	771.093805	496	413.709504	568.4179	416.457143	400
1600	887.906964	548.023599	422.071584	573.550859	437.55814	400.0
1700	993.456911	606.010638	420.628684	571.605276	526.639004	476.0
...	...	...	...	...	...	...
2004	20199.2207	6942.136596	1558.099461	4661.517477	6063.068969	28807.845958
2005	20522.238008	7261.721015	1603.686517	4900.563281	6265.525702	29415.399334
2006	21087.304789	7730.09757	1663.531318	5187.253152	6530.533583	29922.741918
2007	21589.011346	8192.881904	1724.226776	5408.383588	6783.869986	30344.425293
2008	21671.774225	8568.967581	1780.265474	5611.198564	6973.134656	30151.80588

69 rows × 6 columns

Ok. Let's plot using the pandas plot function.

# Select some colors
mycolors = ["#9b59b6", "#3498db", "#95a5a6", "#e74c3c", "#34495e", "#2ecc71"]
# Use seaborn to setup a color map to be used by matplotlib
my_cmap = mpl.colors.ListedColormap(sns.color_palette(mycolors).as_hex())

# Set the size of the figure and get a figure and axis object
fig, ax = plt.subplots(figsize=(30,20))
# Plot using the axis ax and colormap my_cmap
gdppc2.loc[1800:].plot(ax=ax, linewidth=8, cmap=my_cmap)
# Change options of axes, legend
ax.tick_params(axis = 'both', which = 'major', labelsize=32)
ax.tick_params(axis = 'both', which = 'minor', labelsize=16)
ax.yaxis.set_major_formatter(mpl.ticker.StrMethodFormatter('{x:,.0f}'))
ax.legend(prop={'size': 40}).set_title("Region", prop = {'size':40})
# Label axes
ax.set_xlabel('Year', fontsize=36)
ax.set_ylabel('GDP per capita (1990 Int\'l US$)', fontsize=36)

Text(0, 0.5, "GDP per capita (1990 Int'l US$)")

fig