0

How can we identify historical weather data which has changed?

Hi,

We are designing a pattern for ingesting your weather data repeatedly into one of our data pipelines. We are using the historical data feature and I'm referring to this knowledge base article which states that historic data can sometimes change for some weeks or even a few months after it has originally been received in your systems: https://www.visualcrossing.com/resources/documentation/weather-data/how-historical-weather-data-is-updated/?utm_source=chatgpt.com

We are trying to address the issue of late arriving and corrected data in the context of incremental monthly data loads. 

Are there any watermarks / timestamps available in your data which we could query to identify only data changed since we last queried your API for a certain location (we do so based on UK postcode)? Our current logic simply extracts timestamps since we last extracted (based on event date) but this would miss retrospective corrections of data in your systems, or gaps being filled when more recent data already was extracted.

Thanks for helping me to better understand this,

Kind Regards,

Jess

2 comments

Please sign in to leave a comment.