Facebook with Latestnigeriannews  Twieet with latestnigeriannews  RSS Page Feed
Home  |  All Headlines  |  Punch  |  Thisday  |  Daily Sun  |  Vanguard   |  Guardian  |  The Nation  |  Daily Times  |  Daily Trust  |  Daily Independent
World  |  Sports  |  Technology  |  Entertainment  |  Business  |  Politics  |  Tribune  |  Leadership  |  National Mirror  |  BusinessDay  |  More Channels...

Viewing Mode:

Archive:

  1.     Tool Tips    
  2.    Collapsible   
  3.    Collapsed     
Click to view all Entertainment headlines today

Click to view all Sports headlines today

What is web scraping' Here's what you need to know about the process of collecting automated data from websites, and its uses

Published by Business Insider on Thu, 14 Jan 2021


<p><img src="https://static1.businessinsider.com/image/5fff29c6fe7e140019f7eb30-2400/GettyImages-1187635203.jpg" border="0" alt="software developer analyzing data on laptop tablet desktop" data-mce-source="Cavan Images/Getty Images"></p><p></p><bi-shortcode id="summary-shortcode" data-type="summary-shortcode" class="mceNonEditable" contenteditable="false">Summary List Placement</bi-shortcode><p>Web <a href="https://www.businessinsider.com/category/scraping" target="_blank" rel="noopener">scraping</a> is the name given to the process of extracting structured data from third-party websites. In other words, it's a way to capture specific information from one or more websites without also copying unwanted or unrelated information. It's a common practice that has a lot of potential applications and a murky legal profile.&nbsp;</p><h2><strong>What to know about web scraping</strong></h2><p>Web scraping is usually an automated process, but it doesn't have to be; data can be scraped from websites manually, by humans, though that's slow and inefficient. More commonly, scraping is performed by software designed specifically for this application, generally in two main components. A crawler is a program that browses the internet and indexes the content of interest, and it passes this information onto the scraper.</p><p>The scraper is designed to locate the relevant structured information using markers called data locators. These locators indicate the presence of the data, which the scraper then extracts and stores offline in a spreadsheet or database for processing or analysis.</p><p>One simple example of web scraping: Consider a website that aggregates pricing information for retail products so shoppers can see which retailers have the best prices. A scraper can be programmed to index the product pages at every major retailer, with the scraper then visiting each page and using data locators to zero in just on the price field and ignore all the other data on the pageproduct description, reviews, and so on. The scraper can be run daily to update the webpage with the latest pricing information from around the web.&nbsp;</p><h2><strong>How web scraping is used</strong></h2><p>Because there is an enormous variety of data online, there is a wide variety of applications for web scraping. Here are some of the most common uses:</p><ul><li><strong>Price intelligence</strong>: Like the example above, many web scrapers are designed to monitor prices from retail sites. Retailers might use this to monitor prices at competitor sites, or the data might be used for competitive analysis, monitoring trends, or as a service to other users.</li><li><strong style="color: #222222;">Real estate</strong>: Similarly, web scrapers commonly target real estate sites to monitor rental and sale prices, appraise property values in a given region, and conduct market analysis.</li><li><strong style="color: #222222;">Lead generation</strong>: Marketers commonly use web scraping to generate leads by scraping structured data from websites like LinkedIn.</li><li><strong style="color: #222222;">Sentiment analysis</strong>: Brands even use web scraping to understand how their products and services are being talked about online. Companies can collect data that mentions their name from social media sites like Facebook and Twitter.&nbsp;</li></ul><h2><strong>The legality of web scraping</strong></h2><p>There's no easy answer to the question of web scraping's legality. This technology has had a number of legal challenges dating back to 2000, when online auction site eBay filed an injunction (which was granted by the court) against a site called <a href="https://money.cnn.com/2000/04/18/technology/ebay/" target="_blank" rel="noopener">Bidder's Edge for scraping its auction data</a>.&nbsp;</p><p>In the years since, there have been a number of additional challenges to web scraping, but in 2017 <a href="https://www.businessinsider.com/linkedin-loses-appeal-in-suit-against-data-scraping-startup-2019-9" target="_blank" rel="noopener">LinkedIn lost a suit against a business that was scraping its content</a>. With some precedent in the courts both for and against web scraping, it's currently a common practice across the internet.&nbsp;</p><h2><strong>Related coverage from&nbsp;<a href="https://www.businessinsider.com/tech-reference" target="_blank" rel="noopener" data-analytics-post-depth="100">Tech Reference</a>:</strong></h2><ul><li><h3><a href="https://www.businessinsider.com/what-is-streaming" target="_blank" rel="noopener" data-analytics-post-depth="100">The beginner's guide to streaming, including how it works, the pros and cons, and more<br></a></h3></li><li><h3><a href="https://www.businessinsider.com/what-is-lidar" target="_blank" rel="noopener" data-analytics-post-depth="100">What is LiDAR' How everyday devices use lasers to scan your environment</a></h3></li><li><h3><a href="https://www.businessinsider.com/what-is-augmented-reality" target="_blank" rel="noopener" data-analytics-post-depth="100">What is augmented reality' Here's what you need to know about the 3D technology</a></h3></li><li><h3><a href="https://www.businessinsider.com/what-is-machine-learning" target="_blank" rel="noopener">What is machine learning' Here's what you need to know about the branch of artificial intelligence and its common applications</a></h3></li><li><h3><a href="https://www.businessinsider.com/what-is-net-neutrality" target="_blank" rel="noopener" data-analytics-post-depth="100">What is net neutrality' Here's what you need to know about the open internet concept</a></h3></li></ul><p><strong>SEE ALSO:&nbsp;<a href="https://www.businessinsider.com/best-all-in-one-pc'tht" >The best all-in-one PCs you can buy</a></strong></p><p><a href="https://www.businessinsider.com/what-is-web-scraping#comments">Join the conversation about this story &#187;</a></p> <p>NOW WATCH: <a href="https://www.businessinsider.com/what-its-like-to-do-your-own-taxes-for-the-very-first-time-2018-2">July 15 is Tax Dayhere's what it's like to do your own taxes for the very first time</a></p>
Click here to read full news..

All Channels Nigerian Dailies: Punch  |  Vanguard   |  The Nation  |  Thisday  |  Daily Sun  |  Guardian  |  Daily Times  |  Daily Trust  |  Daily Independent  |   The Herald  |  Tribune  |  Leadership  |  National Mirror  |  BusinessDay  |  New Telegraph  |  Peoples Daily  |  Blueprint  |  Nigerian Pilot  |  Sahara Reporters  |  Premium Times  |  The Cable  |  PM News  |  APO Africa Newsroom

Categories Today: World  |  Sports  |  Technology  |  Entertainment  |  Business  |  Politics  |  Columns  |  All Headlines Today

Entertainment (Local): Linda Ikeji  |  Bella Naija  |  Tori  |  Pulse  |  The NET  |  DailyPost  |  Information Nigeria  |  Gistlover  |  Lailas Blog  |  Miss Petite  |  Olufamous  |  Stella Dimoko Korkus Blog  |  Ynaija  |  All Entertainment News Today

Entertainment (World): TMZ  |  Daily Mail  |  Huffington Post

Sports: Goal  |  African Football  |  Bleacher Report  |  FTBpro  |  Kickoff  |  All Sports Headlines Today

Business & Finance: Nairametrics  |  Nigerian Tenders  |  Business Insider  |  Forbes  |  Entrepreneur  |  The Economist  |  BusinessTech  |  Financial Watch  |  BusinessDay  |  All Business News Headlines Today

Technology (Local): Techpoint  |  TechMoran  |  TechCity  |  Innovation Village  |  IT News Africa  |  Technology Times  |  Technext  |  Techcabal  |  All Technology News Headlines Today

Technology (World): Techcrunch  |  Techmeme  |  Slashdot  |  Wired  |  Hackers News  |  Engadget  |  Pocket Lint  |  The Verge

International Networks:   |  CNN  |  BBC  |  Al Jazeera  |  Yahoo

Forum:   |  Nairaland  |  Naij

Other Links: Home   |  Nigerian Jobs