Spyke

Posts

bootstrappable·Empowering people to do the impossiblebygreengnu

Community Prepping

  • Community

** libraries (KEY) Libraries are the most important building in any community in terms of long term value. When in doubt, always support libraries and fund them (regardless of your politics) and work to minimize laws that negatively impact libraries.

EVERY single community that burned library books or got rid of libraries are dead and it is better to be pissed about them having books you don't like than for you to lose your community and you to ultimately die because of it.

Besides books, some libraries also loan out tools and 3D printers; not to mention some provide free classes and shelter to those in need.

** Churches/Temples Make CERTAIN that everyone who is a member knows that in the event of disaster to go to church and bring supplies. This will enable a quick rebuild in the event of damage but also the preservation of as many lives as possible. Feed everyone (regardless of if they are church members or not) who shows up and protect everyone inside from the elements if possible.

If possible operate a soup kitchen for the homeless and have a library of tools on hand for repair. If the government was kind enough to build a fallout shelter for you, stocking a food pantry in there will supercharge community recovery.

Any church/temple that doesn't help the homeless/poor is going to be a deathtrap when disaster strikes (especially those mega-churches where the minister just enriches themselves).

A humble minister who lives in a small apartment/home and helps the homeless all week long is much more likely to have Jesus in his heart and do good than one who owns a private jet and a mansion in a different zip code.

** invincibility centers (done by governments) The centers are in essence a stopgap while utility crews scramble to fix cut power lines and put water mains back into operation, so basic services can be restored to hospitals and essentials prior to restoring services to homes and businesses.

Basically pick any building that the government owns, turn on the generators, let everyone in and give them beverages (coffee/tea/water), electricity and internet access. If done during winter or a heat wave, you'll save thousands (or more) lives.

** Soup kitchens I strongly recommend you set these up before any disaster occurs (or get coordinated with helping one now); because it will not be possible to convince people to give you all of that food AFTER a disaster if you are not actively feeding other people BEFORE a disaster and routinely asking for additional supplies.

It is generally easier to volunteer with one (to get coordinated) than to setup your own but if you need to setup your own, below are the steps.

Working in a soup kitchen will work against you in the event of an extreme famine as people will get paranoid about food hoarding.

*** Outline your basic plan You'll need to figure out these details first: the number of people you plan to serve daily food safety protocols the person in charge of the menu planning food service protocols for those with allergies and religious restrictions whether the food will be served in a cafeteria or restaurant style

*** Look for a Suitable Location Since you’re catering to the homeless, your ideal location should be near where your intended guests are. Remember that most homeless and underprivileged people don’t have cars. If your soup kitchen is far from where they are, it would be inaccessible for them.

If renting or buying your own place isn’t an option right now, you can also set up a soup kitchen in public places. Religious institutions like churches, mosques, synagogues, and temples are a great option too. They are usually accessible to the public and can hold large crowds. Don’t forget to ask for permission from these places first.

*** Register as a Non-Profit Registering your soup kitchen as a non-profit establishes its legal identity. It exempts you from paying tax and makes it easier to avail of government assistance programs for non-profits.

Each state has different procedures for registering charitable organizations. But it usually involves filing an Articles of Incorporation and applying for federal tax exemption. (which can take up to 2 years) To be sure, visit your state’s charity registration office.

*** Additional Paperwork After registering your soup kitchen as a non-profit, you’ll need to take care of additional paperwork like: insurance coverage for property and liability health insurance for your employees (if any) health and safety permits

*** Raise Funds and Food Donations As you know, soup kitchens run on donations. That’s why fundraising activities are essential in getting the soup kitchen up and running. There are plenty of ways to raise funds and donations for a soup kitchen. You can organize a food drive or fundraising activities like: crowdfunding charity auctions online donations selling customized shirts and merchandise charity walk/run charity raffle

*** Look for Volunteers Aside from donations, non-profits like soup kitchens also rely on volunteers to get things done. That’s why aside from raising funds, you also need to recruit volunteers.

Start with the people you know like friends and relatives. You can also involve people from your church, the local schools, or scout for volunteers online. There are plenty of places you can get volunteers from.

Plus, as I’ve said, it’s a great way of bringing people together to help the less fortunate. That’s why most soup kitchens don’t lack volunteers.

Allowing patrons to be volunteers has to be done carefully. You must be selective. It is not uncommon for patrons, who have been given even modest authority, to abuse it and start ordering people (other patrons, volunteers) around. While many patrons are sincere in their desire to help and are trying to give back, it is best to monitor the person until you are sure that they are helping for the right reasons.

*** Buy Necessary Food Service Equipment Obviously, you won’t be able to serve hot meals if you don’t have the proper food serving equipment.

During your planning stage, you already identified how many people you plan to serve per day and the kinds of food you’ll have on your menu. This should give you an idea of what kitchen equipment and tools to buy.

*** Stock Your Pantry This is one of the most important steps in running a soup kitchen. I mean, you can’t cook hot meals if you have nothing to cook, right?

Aside from the food donations, you can also reach out to food banks to see if they can help. Or if you have lots of restaurants and cafeterias in your area, try to see if they can donate some of their excess supplies. Working with local food pantries is where approximately 3/4 of ALL your food will be coming from. Sharing funds with the local pantries will enable bigger food orders and the dollars going much further.

*** Get the Word Out About Your Soup Kitchen Just like when opening a business, you also need to get the word out about your soup kitchen. You can announce it on social media or print posters and flyers. If you know people from the media, then much better.

"Marketing" your soup kitchen will not only help you reach the people you’re hoping to help. It can also connect you with the people who are interested in your cause.

** Food pantries I strongly recommend you set these up before any disaster occurs (or get coordinated with helping one now); because it will not be possible to convince people to give you all of that food AFTER a disaster if you are not actively feeding other people BEFORE a disaster and routinely asking for additional supplies.

It is generally easier to volunteer with one (to get coordinated) than to setup your own.

They tend to follow one of two organizational styles: Shopping Method: The client goes into the pantry; picks out what s/he needs, and fills his/her own bag/box. Ordering Method: The client receives a list of items available, or can view them from behind a table or counter; s/he will indicate which items s/he wants and how many. Staff/volunteers fill the boxes according to the order.

During disaster your first priority is getting supplies to the soup kitchens; so have good relationships with local soup kitchens (as they are the party which will be receiving your excess inventory). you'll be providing a food buffer to soup kitchens to keep people alive until supplies can be brought in from outside the disaster zone. Ideally you'll be well stocked but with current climate change and economic destruction of the working class; it isn't going to be pretty.

Ideally in good times, you'll just be a warehouse for storing food for soup kitchens but when things start to get tough (economically) you'll slowly switch to supplying basics to the working poor (and later to the working middle class) and after that you'll be tapped out and just be working hard trying to get additional supplies for the soup kitchens you partnered with.

Need = (number of people living at or below the poverty line) * 234 lbs. of food aid per year

Working in a food pantry will work against you in the event of an extreme famine as people will get paranoid about food hoarding.

And most food pantries are required to store records for 3 years about the people who received food, specifically: Name Address City, State, Zip Code Phone number Number in household Number of adults Number of children

This information is required by the IRS and is used by the government eligibility programs.

Multiple soup kitchens might pool funds with you to do bulk orders and if you can find a super shopper (someone who extreme coupons), you'll always be able raise funds to get additional supplies. As for every $1 of food a person could donate, $100 worth of food could be purchased by an effective food pantry.

Funding the creation of and growth of community gardens can build a good supply of perishable veg/fruits to supplement long term stored supplies.

View original on slrpnk.net
bootstrappable·Empowering people to do the impossiblebygreengnu

CO2 scrubbing

Each human exhales 42grams of CO2 per hour (1.1Kg per day).

*** Chemical reaction

**** Basic

A process commonly used to teach CO2 scrubbing to chemical engineering students involves sodium hydroxide (NaOH). The reaction is exothermic and is as follows: CO2 (g) + 2 NaOH (aq) → Na2CO3 (aq) + H2O (l)

The aqueous sodium carbonate is subsequently treated with calcium hydroxide (Ca(OH)2) to precipitate out calcium carbonate (CaCO3): Na2CO3 (aq) + Ca(OH)2 (aq) → 2 NaOH (aq) + CaCO3 (s)

Disclaimer:

This is usually done with much higher concentrations of CO2 (e.g. >10%) than you would find in the atmosphere (~400 ppm). The point of scrubbing in a chemical plant is to reduce emissions to a level that is still much higher than atmospheric CO2 levels but lower than governmental standards. I suspect that you would not see a noticeable change in ambient CO2 levels from putting something like this in a place with such (relatively) low CO2 to begin with. There is a very low driving force to push gaseous CO2 into the liquid and solvate it when the partial pressure of CO2 is low.

There exists a unique relationship between the partial pressure of CO2 above a solution, the dissolved carbon in the solution, and the pH of the solution.

So if you wanted to see what was going on in a controlled environment with no other species in the solution / that could enter the solution, you could just use a pH probe to see how much carbon was pushed into the solution.

Calcium hydroxide can be produced when quicklime (CaO) is mixed with water (H20). quicklime is usually made by the thermal decomposition of calcium carbonate (CaCO3). This is accomplished by heating the material to above 825C (1,517F) CaCO3 (s) → CaO (s) + CO2 (g)

The quicklime is not stable and, when cooled, will spontaneously react with CO2 from the air until, after enough time, it will be completely converted back to calcium carbonate unless slaked with water.

**** Apollo Spacecraft

***** Lithium hydroxide (outputs water)

2 LiOH + CO2 → Li2CO3 + H2O One gram of anhydrous lithium hydroxide can remove 450 cm^3 of carbon dioxide gas.

Lithium hydroxide is often produced industrially from lithium carbonate in a metathesis reaction with calcium hydroxide: Li2CO3 + Ca(OH)2 → 2 LiOH + CaCO3 The initially produced hydrate is dehydrated by heating under vacuum up to 180C.

You will need 1.1Kg per person per day. If you are not going into space, buy calcium hydroxide, it is much cheaper ($125 per ton vs $30 per kilogram)

***** Lithium peroxide (outputs oxygen)

It is prepared by the reaction of hydrogen peroxide and lithium hydroxide. This reaction initially produces lithium hydroperoxide: LiOH + H2O2 → LiOOH + H2O

This lithium hydroperoxide may exist as lithium peroxide monoperoxohydrate trihydrate (Li2O2·H2O2·3H2O). Dehydration of this material gives the anhydrous peroxide salt: 2 LiOOH → Li2O2 + H2O2

Li2O2 decomposes at about 450C to give lithium oxide: 2 Li2O2 → 2 Li2O + O2

And absorbs CO2 in the following: Li2O2 + CO2 → Li2CO3 + 1⁄2 O2

Similar to the reaction of lithium hydroxide with carbon dioxide to release 1 Li2CO3 and 1 H2O, lithium peroxide has high absorption capacity and absorbs more CO2 than does the same weight of lithium hydroxide and offers the bonus of releasing oxygen instead of water.

**** Amine gas treating

The primary application for CO2 scrubbing is for the enclosed atmosphere of nuclear submarines. The technology being involves the use of various amines, e.g. monoethanolamine. Cold solutions of these organic compounds bind CO2, but the binding is reversed at higher temperatures: CO2 + 2 HOCH2CH2NH2 ↔ HOCH2CH2NH+3 + HOCH2CH2NHCO−2

The above reaction has been utilized as a primary part of atmosphere control in nuclear submarines since the late 1950s.

The amine concentration in the absorbent aqueous solution is an important parameter in the design and operation of an amine gas treating process. Depending on which one of the following four amines the unit was designed to use and what gases it was designed to remove, these are some typical amine concentrations, expressed as weight percent of pure amine in the aqueous solution: Monoethanolamine(MEA): About 20% for removing H2S and CO2, and about 32% for removing only CO2. Diethanolamine (DEA): About 20 to 25% removing H2S and CO2 Methyldiethanolamine (MDEA): About 30 to 55% for removing H2S and CO2 Diglycolamine: About 50 % for removing H2S and CO2

Additionally you could use: Diisopropanolamine (DIPA) Aminoethoxyethanol (Diglycolamine) (DGA)

Both H2S and CO2 are acid gases and hence corrosive to carbon steel. However, in an amine treating unit, CO2 is the stronger acid of the two. H2S forms a film of iron sulfide on the surface of the steel that acts to protect the steel. When treating gases with a high percentage of CO2, corrosion inhibitors are often used and that permits the use of higher concentrations of amine in the circulating solution.

MEA and DEA are primary and secondary amines. They are very reactive and can effectively remove a high volume of gas due to a high reaction rate. However, due to stoichiometry, the loading capacity is limited to 0.5 mol CO2 per mole of amine. MEA and DEA also require a large amount of energy to strip the CO2 during regeneration, which can be up to 70% of total operating costs. They are also more corrosive and chemically unstable compared to other amines.

Piperazines have been proposed for carbon capture and storage (CCS) because piperazine protects other amines from degradation. Piperazine can be thermally regenerated through multi-stage flash distillation and other methods after being used in operating temperatures up to 150C and recycled back into the absorption process, providing for higher overall energy performance in amine gas treating processes.

Typical absorption column operating range: 35-50C and 5-205 atm of absolute pressure.

*** Fractional Distillation of Liquid Air

Effectively you need only have a good phase diagram for the gases and control the temperature, pressure or both. But lacking those phase diagrams, the following list of triple points should be sufficient to build an approximate working distillation:

(The below looks best in fix width font in a textfile)

| Substance | Temperature (C) | Pressure (kPa) | % of Air | |----------------------+-----------------+----------------+------------| | Air | -213.40 | 5.265 | 100% | | Nitrogen | -209.97 | 12.6 | 78.084% | | Oxygen | -218.79 | 0.152 | 20.9476% | | Water | 0.01 | 0.611657 | 5% - 0.25% | | Argon | -189.34 | 68.9 | 0.934% | | Carbon dioxide | -56.60 | 517 | 0.04% | | Neon | -248.58 | 43.2 | 0.001818% | | Methane | -182.47 | 11.7 | 0.0002% | | Helium | -271.378 | 30.016 | 0.000524% | | Krypton | -157.39 | 74.12 | 0.000114% | | Hydrogen | -259.31 | 7.04 | 0.00005% | | Xenon | -111.8 | 81.5 | 0.0000087% | | Ozone | -192.55 | 0.00114 | 0.000007% | | Nitrogen Dioxide [0] | -11.25 | 18.728 | 0.000002% | | Iodine | 113.50 | 12.07 | 0.000001% | | Acetylene | -80.7 | 120 | Trace | | Ammonia | -77.75 | 6.076 | Trace | | Arsenic | 820 | 3628 | Trace | | Butane | -138.6 | 0.0007 | Trace | | Carbon (graphite) | 4492 | 10132 | Trace | | Carbon monoxide | -205.05 | 15.37 | Trace | | Chloroform | -97.72 | 0.870 | Trace | | Deuterium | -254.52 | 17.1 | Trace | | Ethane | -183.26 | 0.0008 | Trace | | Ethanol | -123 | 0.00000043 | Trace | | Ethylene | -169.2 | 0.12 | Trace | | Formic acid | 8.25 | 2.2 | Trace | | Hexafluoroethane | -100.07 | 26.60 | Trace | | Hydrogen chloride | -114.19 | 13.9 | Trace | | Isobutane | -159.60 | 0.000019481 | Trace | | Mercury | -39.0 | 0.000000165 | Trace | | Nitric oxide | -163.65 | 21.92 | Trace | | Nitrous oxide | -90.81 | 87.85 | Trace | | Palladium | 1552 | 0.0035 | Trace | | Platinum | 1772 | 0.00020 | Trace | | Radon | -71 | 70 | Trace | | Sulfur dioxide | -75.46 | 1.67 | Trace | | Titanium | 1668 | 0.0053 | Trace | | Uranium hexafluoride | 64.02 | 151.7 | Trace |

[0] numbers are a bit questionable

If you have a random sample of gases, you can separate them by gradually cooling the sample until each component gas liquifies. The liquified compound falls to the bottom of a collection vessel. After all of the liquid has been retrieved, cooling continues until the temperature drops to the boiling point of the next compound and it liquifies. Some compounds, such as carbon dioxide, never liquify. Instead, they turn directly into solids, which are easier to retrieve than liquids. (CO2 turns into a liquid over 516.7575kPa between 31.1C and -56.6C)

Put your sample in a vessel that is slightly warmer at the bottom (-185C) than it is at the top (-190C). Oxygen liquifies at -183C, so it flows out of the flask through a tube in the bottom. Nitrogen turns back into a gas, however, because its boiling point is -196C. It flows out through a tube connected to the top of the flask. Repeat to isolate all other gases.

If you only want to isolate CO2, compress it to between 5.11atm (516.7575kPa) and 72.79atm (7375.447kPa) and cool to -57C or at standard atomospheric pressure cool t0 -78C, CO2 gas will directly convert to solid (dry ice) You can do 750 kg/hr at 11Kw or 80 kg/hr at 3Kw

View original on slrpnk.net
dataengineering·Data Engineeringbygreengnu

The 6 columns essential to a $6B/year database table

When you have a database with 6 Billion dollars (US) flowing through it every year, you need to be able to account and prove exactly every single penny and every single action that occurred. So you better have _A tables for all of the main tables and have these columns to boot.

  1. Create_user_id – Who/what created the record
  2. Create_Dt – When exactly was this record created
  3. Update_user_id – If updated, who updated this record (default null)
  4. Update_Dt – when was it last updated (default null)
  5. Archive_Dt – When can we legally destroy these records
  6. Unique_Trans_id – So that tracing down everything that occurred becomes even easier.

It isn't sexy but it'll be handy if you ever need to trace down things in your database too.

View original on slrpnk.net
dataengineering·Data Engineeringbygreengnu

ARCHIVE_DT or when you can finally delete some shit

Knowing when a record can be disposed of in the future is deeply valuable to keeping database tables clean and containing only useful data.

31-DEC-2999 is pretty common for records that don't have an easily known value.

If you are in an organization that is trying to monetize your data, treat this value as the date when the user will no longer be able to see this record.

If you are in a more sane industry, treat it as the records retention date minus X number of days that match your _A table storage duration or if the record is already in your _A table, the date it will be deleted and no longer recoverable.

You may wish to put some thought to your database backup and rotation schedule so that those records cleared by that date as well but I leave that as an exercise to the reader.

View original on slrpnk.net
dataengineering·Data Engineeringbygreengnu

Reference table design

There are 2 ways of doing reference tables:

unique hand written tables that perfectly match your desired data

or

The RT_ tables pattern mixed with cached views which will give a useful versioned reference table with an effective begin date, meaningful descriptions, version number, the effective end date (If it is set). With the ability to get previous version values if needed, who created the values, when the values were created, who updated the values and when they were updated (And if you follow _A table best practices, all of the previous updates too); not that you would likely need to update the values without doing a version update as well.

Insert in the following order to avoid constraint violations:

RT_TABLE

RT_FIELD_DOMAIN (only need to add entries when creating new reference table views or adding columns to reference tables)

RT_TABLE_FIELD (duplicate old RT_FIELD_DOMAIN values with new table to keep old column names)

RT_FIELD_VALUES (Easiest to do 1 row or column at a time)

Or just insert them all in a single transaction

RT_TABLE design

This is the master reference table for finding what reference tables exist and the versions that exist for them.

| Name            | Null     | Type           |
|-----------------+----------+----------------|
| REF_TABLE_ID    | NOT NULL | NUMBER         |
| TABLE_ID        |          | NUMBER         |
| VERSION         |          | NUMBER         |
| NAME            |          | VARCHAR2(30)   |
| DESCRIPTION     |          | VARCHAR2(255)  |
| COMMENTS        |          | VARCHAR2(255)  |
| STATUS          |          | CHAR(1)        |
| CREATE_USER_ID  | NOT NULL | VARCHAR2(20)   |
| UPDATE_USER_ID  |          | VARCHAR2(20)   |
| CREATE_DT       | NOT NULL | DATE           |
| UPDATE_DT       |          | DATE           |
| UNIQUE_TRANS_ID | NOT NULL | NUMBER         |
| EFF_BEGIN_DT    | NOT NULL | DATE           |
| EFF_END_DT      |          | DATE           |
| ARCHIVE_DT      | NOT NULL | DATE           | 

REF_TABLE_ID is the primary key

RT_FIELD_VALUES design

The actual reference table values

| Name               | Null     | Type          |
|--------------------+----------+---------------|
| REF_TABLE_FIELD_ID | NOT NULL | NUMBER        |
| FIELD_ROW_ID       | NOT NULL | NUMBER        |
| FIELD_VALUE        |          | VARCHAR2(255) |
| CREATE_USER_ID     | NOT NULL | VARCHAR2(20)  |
| UPDATE_USER_ID     |          | VARCHAR2(20)  |
| CREATE_DT          | NOT NULL | DATE          |
| UPDATE_DT          |          | DATE          |
| UNIQUE_TRANS_ID    | NOT NULL | NUMBER        |
| ARCHIVE_DT         | NOT NULL | DATE          |

REF_TABLE_FIELD_ID has a foreign key with RT_TABLE_FIELD.REF_TABLE_FIELD_ID FIELD_ROW_ID a sequence value used for all entries on a row

RT_TABLE_FIELD design

This is the glue table for all of the reference tables

| Name               | Null     | Type          |
|--------------------+----------+---------------|
| REF_TABLE_FIELD_ID | NOT NULL | NUMBER        |
| REF_TABLE_ID       | NOT NULL | NUMBER        |
| FIELD_ID           | NOT NULL | NUMBER        |
| CREATE_USER_ID     | NOT NULL | VARCHAR2(20)  |
| UPDATE_USER_ID     |          | VARCHAR2(20)  |
| CREATE_DT          | NOT NULL | DATE          |
| UPDATE_DT          |          | DATE          |
| UNIQUE_TRANS_ID    | NOT NULL | NUMBER        |
| ARCHIVE_DT         | NOT NULL | DATE          |

REF_TABLE_FIELD_ID is the primary key (sequence or uuid) REF_TABLE_ID is a foreign key to RT_TABLE.REF_TABLE_ID FIELD_ID is a foreign key to RT_FIELD_DOMAIN.FIELD_ID

RT_FIELD_DOMAIN design

The actual column names for the reference tables

| Name            | Null     | Type          |
|-----------------+----------+---------------|
| FIELD_ID        | NOT NULL | NUMBER        |
| NAME            |          | VARCHAR2(50)  |
| DATA_TYPE       |          | CHAR(1)       |
| MAX_LENGTH      |          | NUMBER(5)     |
| NULLS_ALLOWED   |          | CHAR(1)       |
| CREATE_USER_ID  | NOT NULL | VARCHAR2(20)  |
| UPDATE_USER_ID  |          | VARCHAR2(20)  |
| CREATE_DT       | NOT NULL | DATE          |
| UPDATE_DT       |          | DATE          |
| UNIQUE_TRANS_ID | NOT NULL | NUMBER        |
| ARCHIVE_DT      | NOT NULL | DATE          |

FIELD_ID is the primary key (sequence or uuid)

RT_ALL_MV design

The master query behind all of the reference tables (keep it cached)

CREATE VIEW IF NOT EXISTS RT_ALL_MV AS
SELECT
   A.NAME AS TABLENAME
  ,A.VERSION AS VERSION
  ,D.FIELD_ID AS FIELDID
  ,A.EFF_BEGIN_DT AS EFFBEGDATE
  ,A.EFF_END_DT AS EFFENDDATE
  ,B.FIELD_ROW_ID AS ROW_ID
  ,D.NAME AS COLUMNNAME
  ,B.FIELD_VALUE AS COLUMNVALUE
FROM
   RT_TABLE A
  ,RT_FIELD_VALUES B
  ,RT_TABLE_FIELD C
  ,RT_FIELD_DOMAIN D
WHERE
  A.REF_TABLE_ID = C.REF_TABLE_ID                AND
  B.REF_TABLE_FIELD_ID = C.REF_TABLE_FIELD_ID    AND
  C.FIELD_ID = D.FIELD_ID;

Example RT_ view

Current values can be just: SELECT * FROM RT_example_MV; For figuring out previous values or making a view:

For sqls that support DECODE

SELECT
   MAX(DECODE(COLUMNNAME, 'CODE', COLUMNVALUE)) AS CODE
  ,MAX(DECODE(COLUMNNAME, 'DESCRIPTION', COLUMNVALUE)) AS DESCRIPTION
  ,MAX(VERSION) AS VERSION
  ,MAX(EFFBEGDATE) AS EFF_BEGIN_DT
  ,MAX(EFFENDDATE) AS EFF_END_DT
FROM FROM RT_ALL_MV
WHERE
  TABLENAME LIKE '%STATUS_IND%' AND VERSION=3
GROUP BY ROW_ID
ORDER BY CODE;

For sqls without

SELECT
   MAX(CASE COLUMNNAME WHEN 'Code' THEN COLUMNVALUE END) AS 'Code'
  ,MAX(CASE COLUMNNAME WHEN 'S0_Rate' THEN COLUMNVALUE END) AS 'S0 Rate'
  ,MAX(CASE COLUMNNAME WHEN 'S1_Rate' THEN COLUMNVALUE END) AS 'S1 Rate'
  ,MAX(VERSION) AS VERSION
  ,MAX(EFFBEGDATE) AS EFF_BEGIN_DT
  ,MAX(EFFENDDATE) AS EFF_END_DT
FROM RT_ALL_MV
WHERE
  TABLENAME LIKE '%example%' AND VERSION=1
GROUP BY ROW_ID
ORDER BY CODE;
View original on slrpnk.net
dataengineering·Data Engineeringbygreengnu

Rate histories or cleanly storing history

The HIST_NAV_IND column:

When you want a history of values (such as ratings) in the main table for some business requirement, add this column and use the following values:

S => When you have only 1 record

F => The first record when you have more than 1 record

P => The current primary record when you have more than 1

M => The previous P records that have been surpassed.

The EFF_BEGIN_DT and EFF_END_DT columns:

In case you might need to do reprocessing of old records you will want an easy way to figure out which rate history that you would want to use; EFF_BEGIN_DT and EFF_END_DT make that simple.

EFF_BEGIN_DT is always set in every record (generally it should match the create date but there are business reasons why you want it separate)

EFF_END_DT should be NULL for the current primary record (unless you are organized enough to always know the future rate change date in advance [unlikely]) and should always be set for the M and F records to the day [or hour, minute or second] prior to the EFF_BEGIN_DT of the new P record. The EFF_END_DT of one record should never overlap with the EFF_BEGIN_DT of the next and you can use TRUNC("TimeStamp", DATE) to ensure that your select driver will always either get 1 [normally] or zero [They shouldn't have been included] records.

View original on slrpnk.net
dataengineering·Data Engineeringbygreengnu

UNIQUE_TRANS_ID or letting you track what occurred together.

You will find 2 different implementations for this, the first (very wrong) is a unique sequence for every table and it serves the purpose of a HIST_SEQ column.

The second (correct) is a global sequence which will be the same for all records in all tables which are updated by a single transaction. The purpose is to make it trivial to find all records (inserted, updated [ and deleted if using _A tables]) in a single transaction. [You'll want to add an AUDIT_UNIQUE_TRANS_ID column to your _A tables for that linkage]

In simple environments this can be just a simple sequence and in more advanced environments this can be a UUID. The key is it must be unique on every transaction but its value should not be used to provide any information about the order of events in a table (that is the job of a HISTORY_SEQ column).

View original on slrpnk.net
dataengineering·Data Engineeringbygreengnu

HISTORY_SEQ column or sanity checking basic mode

If you might need to store multiple duplicate records or want a sequence number for the order of created/updated records in your table.

This is what you need, the big annoying bit is you need to also update this column on EVERY SINGLE UPDATE to that table and you'll want _A tables if you want to figure out historical ordering of events. And you will be creating a unique sequence for every single table where this column exists. but just shove that functionality in a trigger.

This also would be quite handy if you want a unique key handle for picking which records are being manually deleted and you have the solution when one person updates a record at the same time someone else is trying to delete a record.

View original on slrpnk.net
dataengineering·Data Engineeringbygreengnu

_A tables or how not to accidentally lose your shit

Sometimes called journal or audit tables. _A tables do the following magic trick: you can't screw up or delete your data in a way you can't recover.

In the most simple version possible you take your table foo, duplicate it's structure in a table named foo_A and add 2 columns: audit_dt and audit_user_id. Then you create triggers for update and deletes on the table foo to first write the old values as a new insert in the foo_A table.

Now even if you screw up your select and delete all of the contents of table foo. everything will still be in table foo_A. If you accidentally overwrite everything in foo with garbage data, the good data will still be in foo_A

The application nor any of the users need to know about the _A tables (unless you want to leverage stored procedures instead of triggers to create the _A table entries)

View original on slrpnk.net