Hey there Data Engineers. Want to stop putting out fires and start preventing them? Then it might be time to "shift left." By tackling quality, governance, and security from the get-go, you'll save time, money, and headaches.

If you want to learn more, follow the paywall bypassed link to my latest article. I hope some of you find this useful!

Time and again I see the same questions asked: "Why should I use dbt?" or "I don't understand what value dbt offers". So I thought I'd put together an article that touches on some of the benefits, as well as putting together a step through on setting up a new project (using DuckDB as the database), complete with associated GitHub repo for you to take a look at.

Having used dbt since early 2018, and with my partner being a dbt trainer, I hope that this article is useful for some of you. The link is paywall bypassed.

If you're a Data Engineer, before long you'll be asked to build a real-time pipeline.

In my latest article, I build a real-time pipeline using Kafka, Polars and Delta tables to demonstrate how these can work together. Everything is available to try yourself in the associated GitHub repo. So if you're curious, take a moment to check out this technical post.

How often do you build and edit Entity Relationship Diagrams? If the answer is ‘more often than I’d like’, and you’re fed up with tweaking your diagrams, take <5 minutes to read my latest article on building your diagrams with code. Track their changes in GitHub, have them build as part of your CI/CD pipeline, and even drop them into your dbt docs if you like.

This is a ‘friends and family’ link, so it’ll bypass the usual Medium paywall.

I’m not affiliated to the tool I’ve chosen in any way. Just like how it works.

Let me know yours thoughts!

I’ve written a series of Medium articles on creating a Data Pipeline from scratch, using Polars and DeltaTables. The first (linked) is an overview with link to the GitHub repository and each of the deeper dive articles. I then go into the next level of detail, walking through each component.

The articles are paywalled (it took time to build and document), but the link provided is the ‘family & friends’ link which bypasses the paywall for the Lemmy community.

I hope some of you may find this helpful.

A few years ago, if you'd mentioned Infrastructure-as-Code (IaC) to me, I would've given you a puzzled look. However I'm now on the bandwagon. And to help others understand how it can benefit them, I've pulled together a simple GitHub repo that showcases how Terraform can be used with Snowflake to manage users, roles, warehouses and databases.

The readme hopefully gives anyone who wants to give it a go the ability to step through and see results. I'm sharing this in the hopes that it is useful to some of you.

Hi all,

For those wanting a quick repo to use as a basis to get started, I’ve created jen-ai.

There are full instructions in the readme. Once running you can talk to it, and it will respond.

It’s basic, but a place to start.

The 2x OLED displays are still a work in progress

Hi all,

I'm hoping for some troubleshooting tips. I have a self-build split keyboard using Blackpills.

If I flash either side with 'SPLIT_KEYBOARD = no' in my rules.mk, then both sides work fine (but both working as the left side). If however I set 'SPLIT_KEYBOARD = yes', then both sides stop working.

It's driving me insane!

These are the relevant details of rules.mk:

MCU = STM32F401
BOOTLOADER = stm32-dfu
SPLIT_KEYBOARD = yes
SERIAL_DRIVER = usart

And config.h:

#pragma once
#define HAL_USE_SERIAL TRUE
#define SOFT_SERIAL_PIN A12
#define MASTER_LEFT
#define MATRIX_ROWS 6
#define MATRIX_COLS 6
#define SERIAL_USART_DRIVER SD1
#define SERIAL_USART_TX_PAL_MODE 7

And halconf.h:

#pragma once
#define SERIAL_USB_BUFFERS_SIZE 256
#include_next 

And mcuconf.h:

#pragma once
#include_next 
#undef STM32_SERIAL_USE_USART1
#define STM32_SERIAL_USE_USART1 TRUE

Please help! I've been at this for three days.

Hi all,

I'm hoping for some troubleshooting tips. I have a self-build split keyboard using Blackpills.

If I flash either side with 'SPLIT_KEYBOARD = no' in my rules.mk, then both sides work fine (but both working as the left side). If however I set 'SPLIT_KEYBOARD = yes', then both sides stop working.

It's driving me insane!

These are the relevant details of rules.mk:

MCU = STM32F401
BOOTLOADER = stm32-dfu
SPLIT_KEYBOARD = yes
SERIAL_DRIVER = usart

And config.h:

#pragma once
#define HAL_USE_SERIAL TRUE
#define SOFT_SERIAL_PIN A12
#define MASTER_LEFT
#define MATRIX_ROWS 6
#define MATRIX_COLS 6
#define SERIAL_USART_DRIVER SD1
#define SERIAL_USART_TX_PAL_MODE 7

And halconf.h:

#pragma once
#define SERIAL_USB_BUFFERS_SIZE 256
#include_next 

And mcuconf.h:

#pragma once
#include_next 
#undef STM32_SERIAL_USE_USART1
#define STM32_SERIAL_USE_USART1 TRUE

Please help! I've been at this for two days.

I created a 2d surface that I can perform a linear extrusion on, however the result it obviously a hard edge on the extrusion. I would love to be able to add a bezel - either rounded or at 45 degrees. Is there an easy way?

I’ve got the basics down, but now looking to do something more complex, and unsure if it’s possible. I have imported a flat shape (a pcb), and I can extrude this out. What I want to do however is to rotate it on its axis, and then extrude it down the z-axis (not directly out from the surface). Is this possible, and if so, how?

Is there such a thing as an online pcb tester? As in ‘here is my pcb design, when I connect this, I expect it to register as a circuit on these two pins’

Hey there,

Hoping someone could point me to a PCB that is using a Blackpill and a ST7789V OLED display. I have the above and am not 100% on which pins to wire up. GND is obvious. VCC I’m pointing to 3.3V, then there is:

• CS
• DC
• Reset
• SDA
• SCL

The smaller screens only have SDA & SCL. Can anyone confirm whether the others are optional, mandatory, whether there are specific/preferred pins?

I’m feeling a little lost at the moment, so hoping for some guidance.

FYI this is the specific display: here

Hi all,

TL/DR: Ergo Split 36-key with per-key RGB and 2x 2 inch TFT screens, run by 2x BlackPill. Can QMK handle this?

I'm hoping someone can sanity check this for me before I go printing a PCB only to find there is no software capable of doing what I want without major code involvement.

My plan is for a split ergo keyboard with 36 keys (6 rows of 3 per hand, including the thumb cluster). Each key would have an SK6812-MINI-E RGB backlight, and would be run off a BlackPill controller in both sides. I was going to make the connection using a CAT6 patch cable, understanding I don't need all 8 of the cables, but also that a patch cable clicks into place so a reduced chance of shorting anything by disconnecting the sides while still powered.

The gotcha is that I want to run 2x 2" OLED displays, one on each side. Specifically, they are the ST7789V GMT020-02 from Shenzhen GoldenMorning Electronic Co.,Ltd via AliExpress. I know that QMK has Quantum Painter LVGL Integration, and this should support a display. But can I run two independently of one another, one driven off the first BlackPill and the other off the second?

Feedback would be greatly appreciated on this! I have all of the hardware, other than the PCB and case. Before I spend too much time designing the PCB, I want to at least hear that it should be possible, as opposed to hearing that for a variety of reasons it's an absolute non-starter.

Where a comment has been made on my post, it would be awesome to see which post it was related to. It helps give context when there are 15 replies to 5 different posts since I last checked.

Moore's law is the observation that the number of transistors in an integrated circuit (IC) doubles about every two years.

Is there anything similar for the sophistication of AI, or AGI in particular?

I’ve never been much of a musical person. But my kid was playing with a toy piano the other day, pressing buttons and whatnot. I lay my fingers on the keyboard and thought “hmmm… kinda like the home row”.

So my question to the community: have any of you built musical keyboards? Did you post a blog or guide? At first I was thinking a choc switch with a custom long cap that was held off a pivot point at one end and attached to the switch at the other. But musical keyboards vary sound depending on how you press the note. So maybe Hall effect?

Anyway, keen to hear of any adventures down this road by others?

When I eat chicken, I call it chicken. Chicken wing; chicken drumsticks etc.

When I eat lamb, I call it lamb. Lamb shank; lamb cutlets.

So why do I not eat pig or cow? I eat pork or beef. Is there a reason for that?