📊

Web Scraping

Scheduled data collection from public websites and APIs: lawful, polite and monitored.

Our own product

soge.dk: Danish company and vehicle data

Looking for CVR company data or Danish vehicle information? Before anyone scrapes it, check soge.dk, our own search and API for companies and vehicles in Denmark.

You need data that sits on other websites: product catalogues, competitor prices, public registers, events. Copying it by hand does not scale, and a quick script usually works for a couple of weeks, then quietly returns nothing because the site changed its layout. The hard part of scraping is not the first run but the hundredth.

We build collectors that run on a schedule and tell you when they stop working. First we check whether scraping is needed at all: many sites offer an official API, a sitemap, a product feed or structured data inside the page. When a page has to be fetched, we start with plain HTTP and use a real browser only where the site requires it. Requests are paced and spread over time, and proxies are used to share load, not to hammer a site. Before the first request we go through the legal side with you: what the data is, whether personal data is involved and on what basis, the site's terms and robots.txt, and any database rights in the source.

For Møbler Samling (msdf.dk), a furniture-assembly service, we turned the IKEA and JYSK catalogues into 5,937 product pages and 589 categories, each showing the assembly price for that item. IKEA's public search API and sitemaps meant no HTML scraping was needed; for JYSK we read the structured product data embedded in each page. While preparing a refresh, a dry run reported 5,829 products as removed. It was wrong: IKEA had changed its product ID scheme. Acting on that list would have deleted live, indexed pages, which is why every change from a new run goes through a dry run first.

Other collectors we run: Danish company data from the official CVR distribution for uahub.dk, where the source rejects scroll queries, so we page through it by CVR number; a US car-auction data project tracking hundreds of thousands of lots and live bids; and an events feed for Copenhagen, where each source has a stored fingerprint of its shape, so a source that suddenly returns no events or loses its coordinates triggers an alert instead of a quiet zero.

Data collection starts at 2,000 DKK. The price depends on the number of sources, how they are protected, how often they must refresh and whether you want the data cleaned and loaded into a database. Running costs are a small server and, for some sources, a proxy subscription. Scraping is not the right choice when the owner sells an affordable API or data licence, because buying is then cheaper and safer, or when the data is personal information you have no basis to process.

Key Features

Source check before code

For every source we look for an official API, feed, sitemap or JSON-LD before writing a scraper. It is faster, breaks less often and puts less load on the site.

HTTP first, browser when needed

Plain HTTP requests handle most pages. A real browser, Playwright or a stealth browser through Scrapling, is used only for pages that cannot be read without JavaScript.

Polite rate limits

Requests are paced, retried with back-off and scheduled outside the site's busy hours. When a site clearly says no, we stop and look for another route instead of escalating.

Breakage detection

Each run records which parsing path worked and how complete the records were. Zero results, missing fields or a changed layout raise an alert within the day.

Clean, deduplicated output

Records are validated and deduplicated, then delivered as CSV, JSON or straight into your database, with the source URL and collection time on every row.

Dry run before changes

Before data from a new run changes anything on your site or in your systems, you see a diff of what would change, and suspicious numbers are checked by hand.

Why Choose This Service

⚡
Data that stays current

Collectors run on a schedule, so prices and catalogues update without anyone copying them.

⚡
Breakage surfaces within a day

A source that fails quietly raises an alert instead of filling your reports with zeros.

⚡
Less legal guesswork

Terms, personal data and database rights are reviewed before the first request, not after a complaint.

⚡
Output ready to use

You receive structured, validated rows instead of raw HTML you have to clean yourself.

How We Deliver

Our proven process ensures quality at every step

1

Source review

We list the sources, look for APIs and feeds, read the terms and test a few pages by hand. You get a short note on feasibility and on the legal points.

2

Prototype on a sample

A collector for one source runs on a small sample, and we compare the result with the live site field by field before building the rest.

3

Scale and schedule

All sources are added with rate limits, retries and proxies only where justified. Runs are scheduled at hours that suit both you and the source.

4

Monitor and maintain

Shape fingerprints, completeness checks and alerts go live together with the collector. Fixes after a site redesign can be covered by project support.

Frequently Asked Questions

Is web scraping legal in Denmark?

Collecting publicly available data is often lawful, but not always. It depends on whether personal data is involved and on what legal basis, on the site's terms and on database rights in the source material. We go through this with you before building, and if the answer is no, we suggest alternatives such as an official API or a data licence.

What about sites with bot protection?

We first look for a gentler route: an API, a feed, a sitemap or structured data. If a browser is needed, we use one and keep request rates low. We do not solve CAPTCHAs on your behalf, and we do not collect content behind a login you are not entitled to use.

How do you know a collector has stopped working?

Every run stores a fingerprint: how many records came back, which fields were filled and which parsing path worked. When that changes sharply, an alert goes out. The baseline is updated only after a good run, so a broken source cannot make zero look normal.

Can you deliver the data into our own system?

Yes. We deliver CSV or JSON files, push the records to an API, or load them into a database with deduplication and a change history. Loading the data into your shop or CRM is often combined with a small integration, which we quote separately.

Do we need proxies?

Not for most sources. Proxies help when a site limits requests per IP address or shows different content per country. When they are needed, the proxy subscription is part of the running cost we quote, so there are no surprises on the first invoice.

Ready to Cross the Bridge?

Let's build your lightning bridge to digital transformation. From first consultation to live deployment — we move fast.