Forfatter Mads Chr. Olesen

21 okt

0 Comments

Giv mig nu bare elprisen.somjson.dk! (en historie om det måske værste datasæt i åbne energidata)

Af

Der findes rigtig meget åbent data om det danske energi system hos Energi Data Service, f.x. spot prisen på elektricitet og CO2 prognoser per kWh. Det er dog overordentligt svært at finde den samlede pris man betaler som forbruger per kWh, pga. det uigennemskuelige datasæt over tariffer og priser. I frustration kom udbruddet: “Giv mig nu bare elprisen.somjson.dk” der nemt summerer alle priser og afgiter per elselskab, er open source og uden yderligere dikke-darer.

Hvad koster strøm i Danmark?

For en forbruger i Danmark er strømprisen en sum af forskellige priser og afgifter, nogle faste, nogle dynamiske:

  • El-afgiften fastsat ved lov til 0,761 kr per kWh.
  • Energinet’s eltariffer: Nettarif på 0,074 kr per kWh (år 2024), og systemtarif på 0,051 kr per kWh (år 2024).
  • Netselskabstarif fra forsyningsselskabet (N1, Radius, etc.): denne varierer per time og per sæson for at forsøge at incentivere til at udligne forbruget så der ikke skal investeres i nye og større elkabler, og er indkodet i datasættet over tariffer og priser. Private forbrugere betaler C-tarif.
  • Spot-prisen: er den anden variable, og denne varierer ud fra udbud og efterspørgsel.
  • Moms: 25% lagt til summen af ovenstående

Dette er alle de uundgåelige priser og afgifter, der kan regnes ud udelukkende fra adressen. Derudover kommer så det “frie elmarked”, hvor der skal betales til et elselskab: typisk månedsabonnement og et tillæg til spot-prisen.

Tariffer og priser – det værste åbne datasæt?

Som om det ikke er uoverskueligt nok i sig selv, bliver vi nødt til at snakke om datasættet Datahub Price List. Der er flere åbenlyse problemer med det:

  • Der er flere felter man burde filtrere efter, men hvor der ikke findes en udtømmende liste over værdier, f.x. netselskab “ChargeOwner”. Det bedste man kan gøre er at downloade, hvor man så løber ind i at download kun giver 100.000 rækker – og datasættet er fuldt på over 300.000 rækker.
  • ChargeTypeCode er per selskab – og uden systematik. Så for hvert enkelt selskab skal man finde ud af hvilken priskode de bruger for C-tariffen. Og hvad når det ændrer sig?
  • ValidTo kan være udeladt og dermed et open ended interval, og prisen gælder så indtil data retroaktivt ændres. Det betyder også at man ikke kan filtrere på datoer, da prisen på 1. april kan være en række der har en ValidFrom 1. januar (eller tidligere).
  • Price1-24: dette er selve timetarif-priserne. Hvis en pris ikke er udfyldt gælder Price1 – hvorfor I alverden dog tilføje den ekstra kompleksitet!?!?!
  • For ikke at tale om tidszoner: man må antage (det er ikke dokumenteret) at alle datoer og timetal er angivet i hvad end tid der er gældende i Danmark på pågældende dato. Dette giver så problemer ved skift fra sommer-/normal-tid hvor en time gentages eller udelades: hvilken timesats bør bruges i et døgn der har 23 eller 25 timer?

Giv mig nu bare elprisen.somjson.dk!

Efter at have regnet de fleste af de ovenstående problemer ud, skrev jeg en API-proxy der udstiller det API man i virkeligheden vil have: givet en dato, og et elselskab (eller en adresse), returnerer den prisen time for time som et JSON dokument. Prisen er typisk tilgængelig fra kl. 13 dagen før. Som bonus får man også CO2 udledningen med, hvis den er tilgængelig (typisk kl. 15 dagen før). Det er implementeret som en ren API proxy, dvs. det er ren omskrivning af input og data og ikke andet.

Det hele er open source, men der kører en version på elprisen.somjson.dk som frit kan benyttes.

Alternativer

Der findes andre API’er der opfylder forskellige use-cases:

  • Min Strøm API er rigtig modent og har egen forecast model der kan forecaste 7 dage frem, før priserne er låst på Energi Data Service. Kræver en API nøgle og er uden kildekode
  • HomeAssistant energidataservice virker kun med Home Assistant, men fungerer på samme måde mod Energi Data Service.
  • Strømligning API kan bruges til at udregne priser baseret på historisk forbrugsdata. Kan dog også bruges til at hente de forecastede priser. Med rate limiting, og uden kildekode.
  • Carnot har også et åbent API og egen forecast model. Kræver API nøgle, og er uden kildekode.

Gemt under: Extern, HAL9k

Tags: ,

09 okt

0 Comments

World’s Longest Multi-Cutter Blade: 30 cm

Af

I had a need for an extra-extra long multi-cutter blade, so we made one in Hal9k. Until proven otherwise, we hereby claim it to be the world’s longest, at about 30 cm from rotation point to the cutting edge.

Converting Starlock-MAX to Starlock-Plus

Initially I thought I could get away with just a 80mm long Bosch MAIZ 32 APB which seems to be the longest commercially available. First problem was that my multi-cutter was only “Starlock-Plus” and this blade is Starlock-MAX. Turns out, you can easily get around that: just drill up the hole to 10mm, and it fits like a glove, at least on the Starlock-Plus Bosch GOP 18V-28.
Drilling Starlock-MAX to 10mm makes it a Starlock-Plus.

World record time

But as it turns out, I needed more length, and anything worth doing is worth overkilling! So sacrificing an old worn-out blade by welding on some 2mm steel plate provided a good base that would still attach to the multi-cutter. First attempt was just attaching the blade with two 2mm screws, as these are the largest that will fit in the star’s spikes and thereby prevent rotation. Initial testing:
So next solution was to beef up with a central 8mm bolt instead.
This worked much better if torqued enough (read: all you possibly can!), test-run went great after the initial oscillations:
And ultimately the cut in the tight corner was made, one-handedly in order to be able to film:
Great success! This should not be considered safe, and several warranties were probably voided, but it got the job done.

Gemt under: Extern, HAL9k

Tags: ,

15 feb

0 Comments

Reparation af Nordlux IP S12 badeværelseslampe der ikke lyser længere

Af

Denne badeværelseslampe er udgået af produktion, og pga. monteringen og at man ofte har mere end én er det noget træls at skulle udskifte – det giver ihvertfald en del skrot uden grund. Heldigvis er konstruktionen super simpel: det er udelukkende en LED driver (230V AC til 24V DC) og en LED.

Lad os starte med det nemme: LED-driveren er direkte tilgængelig bagfra, og med lidt forsigtighed kan spændingen udmåles. I dette tilfælde var der ca. 24V DC, og det er jo fint indenfor specifikationen.

Selve LED’en er lidt sværere at komme til: fronten af glasset skal drejes af via de to huller deri. Jeg brugte en låseringstang af ca. korrekt dimension, med lidt forsigtighed. Lidt ridser gør nok ikke det store når lyset skinner. LED’en kan nu loddes af.

En ny LED kan købes for ca. 10 kr, f.x. på AliExpress. Det rigtige søgterm er måske “Bridgelux 2020 COB LED”, jeg endte med en 7W i Warm White (3000 Kelvin).

Efter lidt fidlen og lodden er den nye LED monteret, og kan testes. Stor succes!

Gemt under: Extern, HAL9k

Tags: ,

28 okt

0 Comments

Fantus-button part 2: the physical button build and the network communication

Af

First part of this series is here, covering the reverse engineering of the DRTV Chromecast App.

I wanted the physical appearance to be extremely minimalistic, with slight references to various cubes from videogames. Because it is a remote control, it of course has to be wireless and battery-powered.

The box is lasercut from 6 mm MDF, and with a giant red arcade button on top with a red LED inside.

The electronics inside is a battery-powered Wemos D1, along with 4 x 18650 Lithium battery cells. After some experimentation on the response time, which is primarily dominated by the time it takes to reconnect to the WiFi network, I initially only used “light sleep”. This resulted in a battery time of just over a week, which is okay, but not great.

In order to preserve battery deep sleep would be really nice. The problem is deep sleep on the Wemos can only be interrupted by a reset. The idea was to use a MOSFET (in this case an N-channel logic level mosfet, IRFZ44N) for the Wemos to be able to select whether a press of the button should reset it, or it should just register on a pin as normal.

This circuit allows RST to be pulled low by the button, as long as D0 is high. Luckily, D0 is high during deep sleep, so as long as the Arduino code keeps D0 low button presses will not reset — but can still be registered by reading pin D1.

This works out “responsively enough” because the initial start has some delay due to the Chromecast initializing the app and loading media. Any subsequent button presses within the 30 seconds the Arduino stays awake are instant though. With this setup the battery life is not a problem – I’ve only had to charge it once. As a bonus feature/bug whenever the battery gets low the Wemos will trigger a bit sporadically: this causes “Fantus-bombing” where Fantus will just randomly start; quite quickly thereafter the Fantus-button is being charged 😉

The Wemos itself is not powerful enough to do all the pyChromecast communication needed, so I setup a small Raspberry Pi to handle that part. Since I didn’t want to spend too much time and effort setting up the communication between them, I ended up using a trick from my youth: UDP broadcasting. Because UDP is datagram-oriented you can send a UDP packet to the broadcast address (255.255.255.255) and then it will be received by all hosts on the local area network: no configuration needed. In Arduino code it looks like:

  Udp.begin(31337);
Udp.beginPacket("255.255.255.255", 31337);
Udp.write("emergency-button\n");
Udp.endPacket();

(Full Arduino code available here.)

At this point I had a UDP packet that I could receive on the Raspberry Pi, and it was just a matter of writing a small server program to listen, receive and process those UDP commands. However, at this point a thought entered my mind, that derailed the project for a while:

netcat | bash

Why write my own server to parse and execute commands, when Bash is already fully capable of doing exactly that with more flexibility than I could ever dream of? And netcat is perfectly capable of receiving UDP packets? This is a UNIX system, after all, and UNIX is all about combining simple commands in pipelines — each doing one thing well.

The diabolical simplicity of just executing commands directly from the network was a bit too insecure though. This is where Bash Restricted mode enters the project: I wouldn’t rely on it for high security (since it is trying to “enumerate badness“), but by locking down the PATH of commands that are allowed to execute it should be relatively safe from most of the common bypass techniques:

netcat -u -k -l 31337 | PATH=./handlers/ /bin/bash -r

The project was now fully working: press the button, Fantus starts. Press it while Fantus is playing: Fantus pauses. Press it while Fantus is paused: Fantus resumes. The little human was delighted about his new powers over the world, and pressed the button to his hearts content (and his parents slight annoyance at times).

(Full code for handler available here.)

But wouldn’t it be cool if the little human had a (limited) choice in what to view?…

Gemt under: Extern, HAL9k

Tags:

18 jul

0 Comments

Fantus-button part 1: Reverse engineering the DRTV Chromecast App

Af

I want to build a physical giant red button, that when pressed instantly starts a children’s TV-show, in my case Fantus on DRTV using a Chromecast.

The first part of the build is figuring out how to remotely start a specific video on a Chromecast. Initially I thought this would be pretty simple to do from an Arduino, because back in the day you could start a video just using a HTTP request. Very much not so anymore: the Chromecast protocol has evolved into some monster using JSON inside Protobuf over TLS/TCP, with multicast DNS for discovery. Chance of getting that working on a microcontroller is near-zero.

But remote control is possible using e.g. pychromecast which has support for not only the usual app of YouTube, but also a couple of custom ones like BBC. Let’s try and add support for DRTV to pychromecast, starting at the hints given on adding a new app.

Using the netlog-viewer to decode the captured net-export from Chrome, and looking at the unencrypted socket communication, the appId of the DRTV app is easily found.

However, one of the subsequent commands has a lot more customData than I expected, since it should more or less just be the contentId that is needed:

{
  "items": [
    {
      "autoplay": true,
      "customData": {
        "accountToken": {
          "expirationDate": "2022-07-02T00:48:35.391Z",
          "geoLocation": "dk",
          "isCountryVerified": false,
          "isDeviceAbroad": false,
          "isFallbackToken": false,
          "isOptedOut": false,
          "profileId": "c4e0...f3e",
          "refreshable": true,
          "scope": "Catalog",
          "type": "UserAccount",
          "value": "eyJ0eX...Dh8kXg"
        },
        "chainPlayCountdown": 10,
        "profileToken": {
          "expirationDate": "2022-07-02T00:48:35.389Z",
          "geoLocation": "dk",
          "isCountryVerified": false,
          "isDeviceAbroad": false,
          "isFallbackToken": false,
          "isOptedOut": false,
          "profileId": "c4e0a...f3e",
          "refreshable": true,
          "scope": "Catalog",
          "type": "UserProfile",
          "value": "eyJ0eXAi...IkWOU5TA"
        },
        "senderAppVersion": "2.211.33",
        "senderDeviceType": "web_browser",
        "sessionId": "cd84eb44-bce0-495b-ab6a-41ef125b945d",
        "showDebugOverlay": false,
        "userId": ""
      },
      "media": {
        "contentId": "278091",
        "contentType": "video/hls",
        "customData": {
          "accessService": "StandardVideo"
        },
        "streamType": "BUFFERED"
      },
      "preloadTime": 0,
      "startTime": 0
    }
  ],
  "repeatMode": "REPEAT_OFF",
  "requestId": 202,
  "sessionId": "81bdf716-f28a-485b-8dc3-ac4881346f79",
  "startIndex": 0,
  "type": "QUEUE_LOAD"
}

Here I spent a long time trying without any customData, and just using the appId and contentId. Initially it seemed to work!

However, it turned out it only worked if the DRTV Chromecast app was already launched from another device. If launched directly from pychromecast the app would load, show a spinner, and then go back to idle. Here much frustration was spent; I guess the customData is actually needed. And indeed, putting that in works! But where do these tokens come from, and how do we get those tokens from Python?

Using Chrome’s developer tools (F12) on the DRTV page, and then searching globally (CTRL-SHIFT-f) for various terms (“expirationDate”, “customData”, “profileToken”, “accountToken” etc.) revealed some interesting code, that was as semi-readable as any pretty-printed minifyed Javascript. Eventually I found the tokens in local storage:

Using these tokens work really well, and allows starting playback!

Some further exploration proceeded: using the showDebugOverlay flag reveals that the DRTV player is just a rebranded Shaka Player. The autoplay functionality can be disabled by setting chainPlayCountdown to -1, which is honestly a real oversight that it cannot be disabled officially, to not have to rush to stop the playback of the item before the next autoplays.

With all the puzzle pieces ready, I prepared a pull request (still open) to add support for DRTV to pychromecast.

Fantus-button part 2 will follow, detailing the hardware build and network integration with the support from pychromecast.

Gemt under: Extern, HAL9k

Tags: