I helped my friend to download data from the\u00a0DTCC\u2019s Swap Data Repository. I am not familiar with the data and just use this as a programming practice.<\/p>\n
This article gives an introduction to the origin of the data: http:\/\/www.dtcc.com\/news\/2013\/january\/03\/swap-data-repository-real-time<\/a><\/p>\n The Python script will:<\/p>\n As of April 22, 2016, there were around one million historical records. The data seems available from April 6, 2013 and missing sporadically from then on. The Python script will print the bad dates where\u00a0the daily data is not available.<\/p>\n <\/p>\n","protected":false},"excerpt":{"rendered":" I helped my friend to download data from the\u00a0DTCC\u2019s Swap Data Repository. I am not familiar with the data and just use this as a programming practice. This article gives an introduction to the origin of the data: http:\/\/www.dtcc.com\/news\/2013\/january\/03\/swap-data-repository-real-time The … Continue reading \n
import io\r\nimport zipfile\r\nfrom datetime import date\r\n\r\nimport pandas as pd\r\nimport requests\r\n\r\nstart = date(2013, 1, 1)\r\nend = date.today()\r\n\r\nurls = []\r\n\r\nfor i in range(start.toordinal(), end.toordinal()):\r\n datestr = date.fromordinal(i).isoformat().replace('-', '_')\r\n url = ('https:\/\/kgc0418-tdw-data2-0.s3.amazonaws.com\/slices\/CUMULATIVE_CREDITS_' + datestr + '.zip',\r\n 'CUMULATIVE_CREDITS_' + datestr + '.zip')\r\n urls.append(url)\r\n\r\nbadurls = []\r\n\r\ndf = pd.DataFrame()\r\n\r\nfor url in urls:\r\n request = requests.get(url[0])\r\n if not zipfile.is_zipfile(io.BytesIO(request.content)):\r\n print(url[1], 'is non-existent!')\r\n badurls.append(url)\r\n else:\r\n with open(url[1], 'wb') as f:\r\n f.write(request.content)\r\n print(url[1], 'downloaded!')\r\n z = zipfile.ZipFile(io.BytesIO(request.content))\r\n df_ = pd.read_csv(z.open(z.namelist()[0]))\r\n df_['DATE'] = url[1][19:29]\r\n df = df.append(df_, ignore_index=True)\r\n\r\ndf.to_csv('dtcc.csv')\r\n\r\nprint(badurls)<\/pre>\n