Scraping Fanduel Sportsbook For Odds With Beautiful Soup
So i'm very new to web scraping with python and I have a question regarding a project i'm working on. I'm attempting to implement a betting strategy based off of something called t
Solution 1:
So you can get the XHR rquest url by looking at the Dev Tools -> Network tab -> XHR.
It'll return a json structure (which is nested) so you'll have to parse it some way (which I did in the function). It just grabs everything, so if you want specific columns, you can just filter out what you need from the dataframes:
Just remember me when you become a millionaire ;-)...
import requests
import pandas as pd
from pandas.io.json import json_normalize
from functools import reduce
def parse_data(jsonData):
results_df = pd.DataFrame()
for alpha in jsonData['events']:
print ('Gathering %s data: %s @ %s' %(alpha['sportname'],alpha['participantname_away'],alpha['participantname_home']))
alpha_df = json_normalize(alpha).drop('markets',axis=1)
for beta in alpha['markets']:
beta_df = json_normalize(beta).drop('selections',axis=1)
beta_df.columns = [str(col) + '.markets' for col in beta_df.columns]
for theta in beta['selections']:
theta_df = json_normalize(theta)
theta_df.columns = [str(col) + '.selections' for col in theta_df.columns]
temp_df = reduce(lambda left,right: pd.merge(left,right, left_index=True, right_index=True), [alpha_df, beta_df, theta_df])
results_df = results_df.append(temp_df, sort=True).reset_index(drop=True)
return results_df
jsonData_fanduel_nfl = requests.get('https://sportsbook.fanduel.com/cache/psmg/UK/57494.3.json').json()
jsonData_fanduel_nba = requests.get('https://sportsbook.fanduel.com/cache/psmg/UK/55978.3.json').json()
jsonData_fanduel_cbb = requests.get('https://sportsbook.fanduel.com/cache/psmg/UK/53474.3.json').json()
nfl = parse_data(jsonData_fanduel_nfl)
nba = parse_data(jsonData_fanduel_nba)
cbb = parse_data(jsonData_fanduel_cbb)
Output:
print(nba.head(10).to_string())BB.marketsawaypitchernameawayscorelinebig3markettype.marketscompetitorcode.selectionscompetitornumber.selectionscsvavailablebettypes.marketscsvpricetypes.marketsctag.marketscurrentewplaceterms.marketscurrentewreduction.marketscurrenthandicap.selectionscurrentmatchhandicap.marketscurrentmatchhandicap.selectionscurrentpricedown.selectionscurrentpriceup.selectionscurrentscore.selectionsderivativetypes.marketsestimatepricedown.selectionsestimatepriceup.selectionseventlistcodeeventnameeventname.marketsewfactor.selectionsexternaldescriptionfosportinternalorder.marketsgameperiod.marketsgameperiodname.marketshadvalue.selectionshomepitchernamehomescorelineidfobolifestate.selectionsidfoeventidfoevent.marketsidfohadtype.marketsidfomarket.marketsidfomarket.selectionsidfomarketlinetype.marketsidfomarketsubjecttype.marketsidfomarkettype.marketsidfomarkettypeclass.marketsidfomeetingidfoparticipant.selectionsidfoparticipant_awayidfoparticipant_homeidfoselection.selectionsidfoselectionsuspensiontype.selectionsidfosportidfosporttype.marketsidfosporttype.selectionsidfotemplateselection.selectionsinternalorderinternalorder.marketsinternalorder.selectionsiscurrentlistoffer.selectionsisliveismainoutright.marketsismatchismatchhandicaptype.marketsisoutright.marketsisraceistradableistradable.marketsistrapbettingoptionon.marketsisunderover.marketsjockeyname.selectionsjockeysilks.selectionslistcode.selectionslistsubcode.selectionslowerband.selectionsmarketinternaldescription.marketsmtag.marketsname.marketsname.selectionsnontradableon.selectionsnoofmarketsparticipantname_awayparticipantname_homepitcherhacode.marketsprice.selectionsresultpositionfinal.selectionsselectionhashcode.selectionsselexternaldescription.selectionsshortname.selectionssportnamesppricedown.selectionssppriceup.selectionsteletextname.selectionstsstarttsstart.marketsttagleft.marketsttagright.marketsupperband.selections0FalseNoneNoneTPNoneNoneA,C10,C11,C12,C13,C14,C15,C2,C3,C4,C5,C6,C7,C8...CP,SGMNone1.01.0215.5215.5215.523.020.0NoneHP,ONoneNone1771|1772DetroitPistonsAtClevelandCavaliersDetroitPistonsAtClevelandCavaliersNoneDetroitPistonsAtClevelandCavaliers31+GameLinesUNoneNoneN795248.3795248.3OU30740246.330740246.3TOTALNone45773.14778.1 NoneNone1912153.31912141.3106258137.3NoneBASKETBALLBASKETBALLBASKETBALLNone105002 20NoneTrueFalseTrueTrueFalseFalseTrueTrueFalseTrueNoneNoneNoneNoneNoneNoneE795248.3|SCORETotalPoints(half-pointline)UnderNone287DetroitPistonsClevelandCavaliersNone1.869565None29505NoneUnderNBANoneNoneUnder2020-01-07T19:10:00 2020-01-07T19:10:00 NoneNoneNone1FalseNoneNoneTPNoneNoneA,C10,C11,C12,C13,C14,C15,C2,C3,C4,C5,C6,C7,C8...CP,SGMNone1.01.0215.5215.5215.521.020.0NoneHP,ONoneNone1771|1772DetroitPistonsAtClevelandCavaliersDetroitPistonsAtClevelandCavaliersNoneDetroitPistonsAtClevelandCavaliers31+GameLinesONoneNoneN795248.3795248.3OU30740246.330740246.3TOTALNone45773.14778.1 NoneNone1912153.31912141.3106258175.3NoneBASKETBALLBASKETBALLBASKETBALLNone105002 10NoneTrueFalseTrueTrueFalseFalseTrueTrueFalseTrueNoneNoneNoneNoneNoneNoneE795248.3|SCORETotalPoints(half-pointline)OverNone287DetroitPistonsClevelandCavaliersNone1.952381None6650 NoneOverNBANoneNoneOver2020-01-07T19:10:00 2020-01-07T19:10:00 NoneNoneNone2FalseNoneNonePSNoneNoneA,C10,C11,C12,C13,C14,C15,C2,C3,C4,C5,C6,C7,C8...CPNone1.01.033311.010.0NoneH,HPNoneNone1771|1772DetroitPistonsAtClevelandCavaliersDetroitPistonsAtClevelandCavaliersNoneDetroitPistonsAtClevelandCavaliers31+GameLinesHNoneNoneN795248.3795248.3HA30740333.330740333.3SIDENone45778.14777.1 None1.92111e+061912153.31912141.3106259029.3NoneBASKETBALLBASKETBALLBASKETBALLNone1021010 NoneTrueFalseTrueTrueFalseFalseTrueTrueFalseFalseNoneNoneNoneNoneNoneNoneE795248.3|SCORESpreadBettingClevelandCavaliersNone287DetroitPistonsClevelandCavaliersNone1.909091None47283NoneClevelandNBANoneNoneClevelandCavaliers2020-01-07T19:10:00 2020-01-07T19:10:00 NoneNoneNone3FalseNoneNonePSNoneNoneA,C10,C11,C12,C13,C14,C15,C2,C3,C4,C5,C6,C7,C8...CPNone1.01.033311.010.0NoneH,HPNoneNone1771|1772DetroitPistonsAtClevelandCavaliersDetroitPistonsAtClevelandCavaliersNoneDetroitPistonsAtClevelandCavaliers31+GameLinesANoneNoneN795248.3795248.3HA30740333.330740333.3SIDENone45778.14777.1 None1.92113e+061912153.31912141.3106259067.3NoneBASKETBALLBASKETBALLBASKETBALLNone10210NoneTrueFalseTrueTrueFalseFalseTrueTrueFalseFalseNoneNoneNoneNoneNoneNoneE795248.3|SCORESpreadBettingDetroitPistonsNone287DetroitPistonsClevelandCavaliersNone1.909091None987NoneDetroitPNBANoneNoneDetroitPistons2020-01-07T19:10:00 2020-01-07T19:10:00 NoneNoneNone4FalseNoneNonePSNoneNoneA,C10,C11,C12,C13,C14,C15,C2,C3,C4,C5,C6,C7,C8...CP,SGMNone1.01.03.53.53.523.020.0NoneH,HPNoneNone1771|1772DetroitPistonsAtClevelandCavaliersDetroitPistonsAtClevelandCavaliersNoneDetroitPistonsAtClevelandCavaliers31+GameLinesHNoneNoneN795248.3795248.3HA30740357.330740357.3SIDENone45778.14777.1 None1.92111e+061912153.31912141.3106259194.3NoneBASKETBALLBASKETBALLBASKETBALLNone105001 1010 NoneTrueFalseTrueTrueFalseFalseTrueTrueFalseFalseNoneNoneNoneNoneNoneNoneE795248.3|SCORESpread(half-pointline)ClevelandCavaliersNone287DetroitPistonsClevelandCavaliersNone1.869565None44903NoneClevelandNBANoneNoneClevelandCavaliers2020-01-07T19:10:00 2020-01-07T19:10:00 NoneNoneNone5FalseNoneNonePSNoneNoneA,C10,C11,C12,C13,C14,C15,C2,C3,C4,C5,C6,C7,C8...CP,SGMNone1.01.03.53.53.521.020.0NoneH,HPNoneNone1771|1772DetroitPistonsAtClevelandCavaliersDetroitPistonsAtClevelandCavaliersNoneDetroitPistonsAtClevelandCavaliers31+GameLinesANoneNoneN795248.3795248.3HA30740357.330740357.3SIDENone45778.14777.1 None1.92113e+061912153.31912141.3106259237.3NoneBASKETBALLBASKETBALLBASKETBALLNone105001 10NoneTrueFalseTrueTrueFalseFalseTrueTrueFalseFalseNoneNoneNoneNoneNoneNoneE795248.3|SCORESpread(half-pointline)DetroitPistonsNone287DetroitPistonsClevelandCavaliersNone1.952381None42219NoneDetroitPNBANoneNoneDetroitPistons2020-01-07T19:10:00 2020-01-07T19:10:00 NoneNoneNone6TrueNoneNoneMLNoneNoneA,C10,C11,C12,C13,C14,C15,C2,C3,C4,C5,C6,C7,C8...CP,SGMNone1.01.0NoneNoneNone25.033.0NoneNoneNoneNone1771|1772DetroitPistonsAtClevelandCavaliersDetroitPistonsAtClevelandCavaliersNoneDetroitPistonsAtClevelandCavaliers31+GameLinesHNoneNoneN795248.3795248.3HA30740385.330740385.3MONEYNone4937.1 3842.1 None1.92111e+061912153.31912141.3106259265.3NoneBASKETBALLBASKETBALLBASKETBALLNone1011010 NoneTrueFalseTrueFalseFalseFalseTrueTrueFalseFalseNoneNoneNoneNoneNoneNoneE795248.3|SCOREMoneylineClevelandCavaliersNone287DetroitPistonsClevelandCavaliersNone2.320000None23349NoneClevelandNBANoneNoneClevelandCavaliers2020-01-07T19:10:00 2020-01-07T19:10:00 NoneNoneNone7TrueNoneNoneMLNoneNoneA,C10,C11,C12,C13,C14,C15,C2,C3,C4,C5,C6,C7,C8...CP,SGMNone1.01.0NoneNoneNone39.025.0NoneNoneNoneNone1771|1772DetroitPistonsAtClevelandCavaliersDetroitPistonsAtClevelandCavaliersNoneDetroitPistonsAtClevelandCavaliers31+GameLinesANoneNoneN795248.3795248.3HA30740385.330740385.3MONEYNone4937.1 3842.1 None1.92113e+061912153.31912141.3106259271.3NoneBASKETBALLBASKETBALLBASKETBALLNone10110NoneTrueFalseTrueFalseFalseFalseTrueTrueFalseFalseNoneNoneNoneNoneNoneNoneE795248.3|SCOREMoneylineDetroitPistonsNone287DetroitPistonsClevelandCavaliersNone1.641026None16669NoneDetroitPNBANoneNoneDetroitPistons2020-01-07T19:10:00 2020-01-07T19:10:00 NoneNoneNone8FalseNoneNonePSNoneNoneA,C10,C11,C12,C13,C14,C15,C2,C3,C4,C5,C6,C7,C8...CPNone1.01.01.51.51.528.025.0NoneNoneNoneNone1771|1772DetroitPistonsAtClevelandCavaliersDetroitPistonsAtClevelandCavaliersNoneDetroitPistonsAtClevelandCavaliers31-21stHalfHNoneNoneN795248.3795248.3HA30740405.330740405.3SIDENone47598.15303.1 NoneNone1912153.31912141.3106259531.3NoneBASKETBALLBASKETBALLBASKETBALLNone102021010 NoneTrueFalseTrueTrueFalseFalseTrueTrueFalseFalseNoneNoneNoneNoneNoneNoneE795248.3|SCORE1stHalfSpreadClevelandCavaliersNone287DetroitPistonsClevelandCavaliersNone1.892857None30949NoneClevelandNBANoneNoneClevelandCavaliers2020-01-07T19:10:00 2020-01-07T19:10:00 NoneNoneNone9FalseNoneNonePSNoneNoneA,C10,C11,C12,C13,C14,C15,C2,C3,C4,C5,C6,C7,C8...CPNone1.01.01.51.51.527.025.0NoneNoneNoneNone1771|1772DetroitPistonsAtClevelandCavaliersDetroitPistonsAtClevelandCavaliersNoneDetroitPistonsAtClevelandCavaliers31-21stHalfANoneNoneN795248.3795248.3HA30740405.330740405.3SIDENone47598.15303.1 NoneNone1912153.31912141.3106259532.3NoneBASKETBALLBASKETBALLBASKETBALLNone1020210NoneTrueFalseTrueTrueFalseFalseTrueTrueFalseFalseNoneNoneNoneNoneNoneNoneE795248.3|SCORE1stHalfSpreadDetroitPistonsNone287DetroitPistonsClevelandCavaliersNone1.925926None13664NoneDetroitPNBANoneNoneDetroitPistons2020-01-07T19:10:00 2020-01-07T19:10:00 NoneNoneNone
Post a Comment for "Scraping Fanduel Sportsbook For Odds With Beautiful Soup"