:ih ddlmZmZddlZddlmZmZmZmZm Z m Z m Z m Z ddl mZddlmZddlmZddlmZddlmZmZmZmZmZmZdd lmZdd lmZdd l m!Z!dd l"m"Z"eZ#ejHe%Z&d e'de(de'de(fdZ)e#jUdeeejVeddedddedddeejXfdede'de'de-d ee-f d!Z.e#j_de0e jb"ed#d$%ed&d'd(d)*ed#d+%eejVeejXfd,ed-e d.e(d/e'de(def d0Z2e#j_d1ee jb"e d2d3%eejVeejXfd4e defd5Z3e#j_d6ee jb"eejVeejXfd7edefd8Z4e#jkd9e0eejVeejXfd:e'd,edefd;Z6e#jod9e0eejVeejXfd:e'defd<Z8e#jUd9eeejVeejXfd:e'ded=efd>Z9e#j_d?eed#d@%ed&d'd(dA*ed#dB%eejVeejXfd:e'd.e(d/e'de(def dCZ:e#j_dDe0e jv"ed#d@%ed&d'd(dA*ed#dB%eejVeejXfd.e(d/e'de(defdEZe#j_dKe0eejVeejXfd:e'defdLZ?d e'fdMZ@e#j_dNe0eejVeejXfd:e'd-e defdOZAe#jUdPe0eejVeejXfd:e'defdQZBy)R)ListOptionalN) APIRouterDepends HTTPExceptionQuerystatusBackgroundTasks UploadFileFile) AsyncSession)select)deps)Product) ProductCreate ProductUpdateProductResponseProductListResponseBulkProductUploadBulkUploadResponse)ScrapingResponse)ProductService)ScraperServicedatetime product_id headless_modetimeout_secondsenable_discoveryc K ddlm}ddlm}tj d||4d{}|j t tjtj|k(d{}|jj}|s+tjd|d dddd{y|j} |j} t|j } tj d| d |d tj d g} g} ddd d ddd d d}tj d t#j$|j|j|j ||d{\}}| j'|| j'|t)||dd<t)||dd<d|dd<tj dt)|dt)|dtj d t#j.|||||d{}|j1dg}|j1dg}| j'|| j'|t)||dd<t)||dd<d|dd<tj d t)|dt)|dt3}g}| D]v}|j1d"|j1d#|j1d$t5|j1d%dd&f}||vsU|j7||j9|x|j:j=}||_|jAd{tj d'd(tj d)|tj d(tj d*| tj d+t)| tj d,t)|tj d-|tj d.|ddd/|ddd|ddd0tj d1|ddd/|ddd|ddd0tj d(d'dddd{y77s7,7i#t*$rB}dt-||dd<tjdt-|dYd}~0d}~wwxYw7#t*$rB}dt-||dd<tjd!t-|dYd}~d}~wwxYw77#1d{7swYyxYw#t*$r0}tjd2|d3t-|d4Yd}~yd}~wwxYww)5z Background task to perform scraping after product creation. This runs asynchronously without blocking the API response. Executes UNIFIED SCRAPING: Google Search + Shopping Light (same as /run-scraping endpoint) rAsyncSessionLocalruG🚀 [Background] Starting UNIFIED first-time scraping for product ID: Nu❌ [Background] Product z not foundz[Background] Product:  (ID: )zC[Background] Using UNIFIED approach: Google Search + Shopping Lightpendingresults violationsr  google_searchshopping_lightu?[Background] [STEP 1/2] 🔍 Google Search (Organic Results)... product_namebarcodemspproductdbr*r'r( completedr u [Background] ✅ Google Search: results, violations failed - u*[Background] ⚠️ Google Search failed: Fexc_infouJ[Background] [STEP 2/2] 🛒 Google Shopping Light + Registered Vendors...rrrrr+u![Background] ✅ Shopping Light: u+[Background] ⚠️ Shopping Light failed: vendor_namer-r/ scraped_price P================================================================================u<✅ [Background] UNIFIED SCRAPING COMPLETED for product ID: u • Product:  • Total Results: u( • Total Violations (deduplicated): u • Execution Time: u • Google Search: z (z violations)u • Shopping Light: u;⚠️ [Background] UNIFIED scraping failed for product ID : T)!app.db.sessionr"rloggerinfoexecuterrwhereidscalarsfirsterrorr-r.floatr/rsearch_google_serpextendlen Exceptionstrscrape_product_serpgetsetroundaddappendutcnow isoformatlast_execution_timecommit)rrrrr"rr1resultr0product_name_cachedproduct_barcode_cachedproduct_msp_cached all_resultsall_violationssearch_summarygoogle_resultsgoogle_violations google_errshopping_resultshopping_resultsshopping_violations shopping_errseen_violationsdeduped_violationsv violation_keyexecution_time_stres =/var/www/html/marco-python-backend/app/api/routes/products.py_perform_background_scrapingrns|oz4% ]^h]ijk$&f 'f '"::fWo&;&;GJJ*2%%&78=@=P/ :@CDU@V/ =s>?R>SS]^abs^t]uvABC KKd f p(6(J(J)%5"/$3 )#$3#6#6y"#E &5&9&9,&K#""#34%%&9:>ABR>S/0;ADEXAY/0>=H/0: ?DT@U?VV`adexay`z{FGH "eO!# # 1EE-(EE.)EE%L!%%3Q7 ! !7#'' 6&--a0 1"1!2! +=G '))+   KK"VH & KKVWaVbc d KK6( $ KK*+>*?@ A KK0[1A0BC D KKB3GYCZB[\ ] KK12D1EF G KK01PQY1Z0[[]^lm|^}H_I^JJTUcdsUtuAUBTCCOP Q KK1.AQ2RS[2\1]]_`noaABKaL`MMWXfgwXxyEXFWGGST U KK6(" &Mf 'f 'f 'Vf '65 m>GJGXV8 V.S/V.W* BV;S5 S2 BS5 V!U>U?B"U!A"VAV!V"D6V V.#V$V.(W*)V.,V/V.2S55 U>7T;5V;UVU V7V V VVV.V+V" V+'V.*W*+V.. W'7&W"W*"W''W*/)response_model)ge d)rrler-z&^(product_name|msp|last_scraped_date)$)regexr1pagelimitsortBysearchcvKtj|||||d{\}}t||||S7w)zN Retrieve a list of products with pagination, sorting, and searching. )rwrxsort_byrzN)itemstotalrwrx)r get_productsr)r1rwrxryrz current_userproductsr~s rmrrsD+77 UF6OHe XUU SSs 979)rp status_codeTz1Run browser in headless mode for initial scraping) description<z8Timeout in seconds for page load during initial scraping)rrrurzADiscover alternative vendors via SERP API during initial scraping product_inbackground_tasksheadlesstimeoutcXKtj||d{}tjd|jd|j d|j t|j |||tjd|j d|j dd d S7w) a Add a new product and automatically trigger scraping in the background. **Returns immediately** (HTTP 201) without waiting for scraping to complete. Scraping happens asynchronously in the background using SERP API. Parameters: - product_in: Product creation data - headless: Whether to run browser in headless mode (default: True) - timeout: Page load timeout in seconds (default: 15) - enable_discovery: Discover alternative vendors via SERP API (default: True) **Response (Immediate):** - message: Product creation success - product_id: ID of created product - scraping_status: "in_progress" - indicates scraping is running in background **Note:** Scraping continues in background. You can check results via GET /products/{id} or monitor logs for completion message: "First-time scraping completed for product ID: {id}" Nu✅ Product created: r#r$)rrrru3📌 Background scraping scheduled for product ID: zCProduct added successfully! Scraping is starting in the background. in_progresszuYou can start working with the product while scraping happens. Check logs or GET /products/{id} for scraping results.)messagerscraping_statusnote)rcreate_productrArBr-rEadd_taskrn)rrrrrr1rr0s rm add_productrs>#11"jAAG KK'(<(<'=VGJJ$?@A )&*>)?@A ofX&6%789 )&1E*F)GHI (  NN^ -h'+ .se}- . vhbM"!"#45l+(#%&:;(#9%   ]%`  /y8PQTUVQWPXYZ  /y3q6(CD N.    @QITX Y==23q6(;   sP:OMCO"BM"%O&BM",AOODOP:O" O+$NO O!$O O OOP7(A P22P77P:z/bulk-upload/json bulk_uploadc ^Ktjdt|jdt j ||jd{}tjddtjdtjd|dtjd |d tjd |d tjd |dtjddt |d|d |d |d|d|dS7حw)aq Bulk upload products from JSON data. **Request body example:** ```json { "products": [ { "reference_id": 15203, "product_name": "COMPOSOR 3 HEPAVESICAL COMPLEX SXXI", "barcode": "8422947152031", "msp": 13.26, "status": true }, { "reference_id": 15205, "product_name": "COMPOSOR 5 SEDANER C", "barcode": "8422947152055", "msp": 17.45, "status": true } ] } ``` **Features:** - Auto-calculates pack prices based on MSP using same discount logic - Validates barcode uniqueness (skips duplicates) - Validates all fields according to ProductCreate schema - Returns detailed error report with row numbers **Response includes:** - total_processed: Number of items in request - successful: Products created - failed: Products with validation errors - skipped_duplicates: Products with duplicate barcodes - errors: Detailed error list u#📦 [BULK UPLOAD JSON] Processing productsNr<r=u ✅ [BULK UPLOAD JSON] Completedrrrrrrrrrrr)rArBrLrrrr)rr1rrYs rmbulk_upload_jsonrts'X KK5c+:N:N6O5PPYZ["66r;;O;OP PF KK"VH  KK24 KK.(9!: ;<= KK%f\&:%;<= KK/&"2!345 KK%f-A&B%CDE KK6("  01,'h!"67hy!  QsAD-D+CD-z/{id}rEcTKtj|||d{ddiS7w)z0 Edit an existing product. (Admin only) NrzProduct updated successfully)rupdate_product)rErr1rs rmrrs1  ' 'B ;;; 5 66>33G  "00 ) F M!="A0A,AA0%A.&A0.A0z /scrape-allc @ Kttjtjdk(}|j |d{}|j j }|sttjdg}d} d} g} tjddtjdt|d tjdd  t|d D]"\} } d} dd l m}|}tjd d tjd| dt|d| jd| j d tjd tjd| j"tjd| j$tjd|t'j(|| j |||d{}|j+dd}|j+dd}tjd| j dtjd|tjd||j+dg}|rTtj-d| jd|D],}tj-d|d d!|d"d#d$|d%d#d.| |z } | |z } |j/||s |j9d{%tjddtjd,tjdtjd-t|tjd.t|tjd/t| tjd0| tjd1| tjdd d2d3t|d t|t|t| | | || r| nd|d4 S77I#t$ro}tj1d&| j d'|j2d()| j/| j | j|j2d*Yd}~d}~wt4$rm}tj1d&| j d't7|d)| j/| j | jt7|d*Yd}~4d}~wwxYw7$#t4$r,}tj-d+t7|Yd}~ud}~wwxYw#|rR |j9d{7w#t4$r+}tj-d+t7|Yd}~wd}~wwxYwwxYw#t4$rO}tj1d5t7|d)ttj:d6t7|d}~wwxYww)7a< Manually trigger scraping for ALL active products. Scrapes EACH product against ALL registered active vendors. For each vendor's website URL: Extract price using search and price detection. Then if discovery enabled: Google search for alternative vendors and scrape them too. This is a SYNCHRONOUS operation - waits for all products to complete before returning. Products are scraped sequentially to avoid browser and connection issues. Parameters: - headless: Whether to run browser in headless mode (default: True) - timeout: Page load timeout in seconds (default: 15) - enable_discovery: Also discover and scrape alternative vendors via Google search (default: True) Returns: - Complete results including all scraped vendors and violations for each product TN%No active products found for scrapingrr r=u(🚀 [SCRAPE ALL] Starting scraping for rr<rqr!────────────────────────────────────────────────────────────────────────────────[ro] 📦 PRODUCT: r#r$ Barcode:  MSP: ₹ Discovery Enabled: r8 scraped_countviolation_count ✅ [RESULT] Product :u • Vendors Scraped: u • Violations Found: r( 🚨 VIOLATIONS FOR  - r9: ₹r:z.3f (MSP: ₹r/ ❌ [ERROR] Product r?Fr6rr-rHWarning closing session: u📊 [SCRAPE ALL] FINAL SUMMARYTotal Products: Successfully Scraped: Failed: zTotal Vendor URLs Scraped: Total Violations Found: r2Scraping completed for ) r rtotal_productssuccessful_productsfailed_productstotal_vendors_scrapedtotal_violationsr failed_listru' ❌ [FATAL ERROR] Scrape All failed: Scraping failed: )rrrDr rCrFrrrrArBrL enumerater@r"r-rEr.r/rrrPrrTrHrrMrNcloser)rrrr1rstmtrYrall_products_results total_scrapedrridxr0fresh_dbr"product_resultrrr(riherls rmscrape_all_productsrs4 '? 4!7 8D::d# #F~~##%H 33:  MO KK$vh  KK:3x=/ST KK6(" ] %h2= MLCH; M<,. b O, auAc(m_4DWEYEYDZZ`ahakak`llmno xj* //@AB .w{{m<= 9:J9KLM(6'D'D&zz%5"*$+ ("!/ 2 2?A F "0"4"45F"J 5gjj\CD 6}oFG 77HIJ,// bA NN%>w?S?S>TTU#VW'|!M2B1C5?I[\_H``jklmrkstwjxxy'z{|.  O3 $++N;$M&nn...w= M@  d6(O$ 57 vh  &s8}o67 ,S1E-F,GHI hs?3456 1-AB ./?.@AB vhbM""05I1J0K9U!(m#&';#<"?3%2 0,.=?4 0  {$D"4!  5gjj\BII;OZ_ `&&")**$+$8$8YY(   5gjj\CF8LW[ \&&")**$+$8$8 V( /$M)B3q6('KLLMM&nn...$M)B3q6('KLLM<   @QITX Y==&s1vh/   s AXP:BX!W7C=Q4P=5C>Q3W8T2 T/ T2D)W9X=Q T, A$R3-U*3 T,?A"T'!U*'T,,U*/T22 U';!U"W"U''W*W.VV VW V< !V7 2W7V< <WW X A XXXz/{id}/scrape-serpz>Also discover alternative vendors via SERP API Google ShoppingcKtj||d{}|jsttjdt j |||||d{}|S7O7w)a Manually trigger scraping for a specific product using SERP API for discovery. Same functionality as /{id}/scrape, but uses SERP API Google Shopping Light Engine instead of Tavily API for discovering alternative vendors. Scrapes the product against all registered active vendors. When discovery is enabled, also searches SERP API Google Shopping for alternative sellers and scrapes their prices. Parameters: - id: Product ID to scrape - headless: Whether to run browser in headless mode (default: True) - timeout: Page load timeout in seconds (default: 15) - enable_discovery: if true, discover and scrape alternative vendors via SERP API (default: true) Nrrr8)rrr rrrrOrs rmtrigger_scraping_serprsu0#44R<>33G  "55 ) F M!=rz/scrape-all-serpc Kttjtjdk(}|j |d{}|j j }|sttjdg}d} d} g} tjddtjdt|d tjdd  t|d D]\\} } d} dd l m}|}tjd d tjd| dt|d| jd| j d tjd tjd| j"tjd| j$tjdtjd|t'j(|| j |||d{}|j+dd}|j+dd}tjd| j dtjd|tjd||j+dg}|rytj-d| jd|D]Q}tj-d |j+d!d"d#|j+d$d%d&|j+d'd%dS| |z } | |z } |j/||sD |j9d{_tjddtjd.tjdtjd/t|tjd0t|tjd1t| tjd2| tjd3| tjd4tjdd d5d6t|d t|t|t| | | || r| nd|d7d8d9 S7E7#t$ro}tj1d(| j d)|j2d*+| j/| j | j|j2d,Yd}~d}~wt4$rm}tj1d(| j d)t7|d+| j/| j | jt7|d,Yd}~Kd}~wwxYw7;#t4$r,}tj-d-t7|Yd}~d}~wwxYw#|rR |j9d{7w#t4$r+}tj-d-t7|Yd}~wd}~wwxYwwxYw#t4$rO}tj1d:t7|d+ttj:d;t7|d}~wwxYww)u • Total Violations Found: r(rrr9Unknownrr:zN/Arr/rr?Fr6rru$📊 [SCRAPE ALL SERP] FINAL SUMMARYrrrzTotal Results: rz7Discovery Method: SERP API Google Shopping Light Enginer2rserp_api_google_shoppingbrowser_with_serp_api_discovery) r rrrrrrrrrdiscovery_method source_typeu, ❌ [FATAL ERROR] Scrape All SERP failed: r)rrrDr rCrFrrrrArBrLrr@r"r-rEr.r/rrOrPrrTrHrrMrNrr)rrrr1rrrYrrrrrrr0rr"rrtotal_violation_countr(rirrls rmscrape_all_products_serprs: '? 4!7 8D::d# #F~~##%H 33:  MO KK$vh  KK?H Nfgh KK6(" a %h2> MLCH< M<,. b O, auAc(m_4DWEYEYDZZ`ahakak`llmno xj* //@AB .w{{m<= PR 9:J9KLM(6'I'I&zz%5"*$+ ("!/ 2 2?A F (6(:(:;Mq(Q% 5gjj\CD 4]ODE =>S=TUV,// bA NN%>w?S?S>TTU#VW'Y!%% y2Q1RRWXYX]X]^motXuWvwABCBGBGHMOTBUAVVW(XYY.  $99 $++N;$M&nn...y> MB  d6(O$ :< vh  &s8}o67 ,S1E-F,GHI hs?3456 om_56 ./?.@AB MO vhbM""05I1J0KKcd!(m#&';#<"?3* 0,.=?4 0 :<  $F"4!  5gjj\BII;OZ_ `&&")**$+$8$8YY(   5gjj\CF8LW[ \&&")**$+$8$8 V( /$M)B3q6('KLLMM&nn...$M)B3q6('KLLMB   Ec!fXNY] ^==&s1vh/   s AY/R BY/!X7DR R D#R-X2VVV EX Y/R U=A$T>V; U=A"U82V;8U==V;V V8 !V3-X3V88X;X?WW WX X "!X XX XX Y,A Y''Y,,Y/z/{id}/search-googlec 0K tj||d{}tjddtjd|jtjddt j |j|j|j||d{\}}tjdt|dt|dd ||j|jt|jt|||t|d d d d S77#t$rt$rO}tjdt|dttj dt|d}~wwxYww)ae Search for a product using Google Search (SERP API - regular search engine). Creates violations in the database following the same schema as SERP Shopping Light API. Parameters: - id: Product ID to search for Returns: - Search results from Google Search (organic_results) - Violations found and stored in database Nr<r=u(🔍 [Google Search] Searching product: r,u,[Google Search] ✓ Search completed. Found r3r4r2r*serp_apiz^Results and violations are stored in database following same schema as SERP Shopping Light API) r rr-r.r/rr'r(r search_enginesourceru❌ Google Search failed: Tr6zSearch failed: r)rrrArBr-rrJr.r/rLrIrrMrHrNr r)rEr1rr0r'r(rls rmsearch_google_productrRsw") &88R@@ bM" >w?S?S>TUV vhbM"%3$E$E --OO  %    B3w<.PZ[^_i[jZkkvwx"#00% \$ #J, t  #A 2    1#a&:T J==$SVH-   sMFD2D-BD2/D00A2%%&78=@=P/ :@CDU@V/ =ABR>S/0;ADEXAY/0>=H/0: 5DG# &EE-(EE.)EE%L!%%3Q7  d?HHSMNN1% &-X__.88:N KK8 E F KK)C $4#5^CL>R S'2G #,;HOO,=G )))+  ubbbL (5 Q>GAx>33G  (  KKFrdKL ; #=A?A=A!A?z/{id}/scraping-statuscKtj||d{}|sttjd||j |j xsd|j|j|jdS7`w)NzProduct not foundridle)rr-r started_at completed_atrH) rrrr HTTP_404_NOT_FOUNDr-rrrrrs rmget_scraping_statusr&s #44R<r@s!ggg/&277    8 $zzzzzzzz zzzC 34t{{+aAraC(.VW ../ TT T T  T SM T5T"S63J3JK4-`ab6pq"45xyt{{+../11%11 1  1  1L1h !2DRXRiRijC-[\t{{+../R R R kR j 1CQWQhQhit{{+../?"??j?DGD)t{{+../ 7  7 7  7* 7wt,t{{+../ 7  7 7- 7GO4t{{+../     5 ^,<=4-KLb6XY"45jkt{{+../ # ### #  #>#L]4V=O=OP4-KLb6XY"45jkt{{+../ I I I I   I QI X 64-KLb6XY"45uvt{{+../ ' ''' '  '7'T &BTBTU4-KLb6XY"45uvt{{+../ P P P P   P VP f "48t{{+../9  9 9 99 xfsfP !$7t{{+../  % 8: #D9t{{+../ :r?