Verification
The numbers on this page are recomputed from the live calculation code every time the site is built. If one of them moved away from its reference, the build would stop and nothing would be published.
Thermocouples against NIST ITS-90
Every value of the NIST SRD 60 tables in 1 °C steps, compared with the reference function used on this site. The tables are rounded to 0.001 mV, so rounding alone can cause up to 0.0005 mV.
| Type | Range | Table values compared | Largest deviation | Source |
|---|---|---|---|---|
| Type K thermocouple | -270 … 1372 °C | 1643 | 0.00050 mV | NIST ITS-90, type K |
| Type J thermocouple | -210 … 1200 °C | 1411 | 0.00050 mV | NIST ITS-90, type J |
| Type T thermocouple | -270 … 400 °C | 671 | 0.00050 mV | NIST ITS-90, type T |
| Type E thermocouple | -270 … 1000 °C | 1271 | 0.00050 mV | NIST ITS-90, type E |
| Type N thermocouple | -270 … 1300 °C | 1571 | 0.00050 mV | NIST ITS-90, type N |
| Type R thermocouple | -50 … 1768.1 °C | 1819 | 0.00050 mV | NIST ITS-90, type R |
| Type S thermocouple | -50 … 1768.1 °C | 1819 | 0.00050 mV | NIST ITS-90, type S |
| Type B thermocouple | 0 … 1820 °C | 1821 | 0.00050 mV | NIST ITS-90, type B |
PT100 against IEC 60751
Values of the IEC 60751 table (0.01 Ω resolution) next to the Callendar–Van Dusen result.
| Temperature (°C) | Published (Ω) | This site (Ω) | Difference (Ω) |
|---|---|---|---|
| -200 | 18.52 | 18.5201 | +0.0001 |
| -100 | 60.26 | 60.2558 | −0.0042 |
| 0 | 100.00 | 100.0000 | +0.0000 |
| 100 | 138.51 | 138.5055 | −0.0045 |
| 200 | 175.86 | 175.8560 | −0.0040 |
| 300 | 212.05 | 212.0515 | +0.0015 |
| 400 | 247.09 | 247.0920 | +0.0020 |
| 500 | 280.98 | 280.9775 | −0.0025 |
| 600 | 313.71 | 313.7080 | −0.0020 |
| 700 | 345.28 | 345.2835 | +0.0035 |
| 800 | 375.70 | 375.7040 | +0.0040 |
| 850 | 390.48 | 390.4811 | +0.0011 |
Nickel RTD against a published DIN 43760 table
311 values of the E+E Elektronik Ni1000 TK6180 table from −60 to 250 °C; largest deviation 0.0050 Ω at a table resolution of 0.01 Ω. Coefficients as printed in the TE Connectivity Ni1000SOT datasheet. Source
Worked reference cases
Each reference value follows from the stated inputs by the formula shown on the tool page and was calculated independently of the site's code.
| Calculator | Inputs | Reference | This site |
|---|---|---|---|
| 4–20 mA converter | 12 mA · 0 … 10 bar | 5.0000 bar | 5.0000 bar |
| 4–20 mA loop power calculator | 24 V · 10.5 V · 22 mA · 250 Ω · 300 m · 0.75 mm² · 20 °C | 7.69728 V | 7.69728 V |
| NTC thermistor calculator | 10 kΩ · β 3950 K · 25 °C → 50 °C | 3588.18 Ω | 3588.18 Ω |
| Load cell output calculator | 2 mV/V · 10 V · 4 × 1000 kg · 1800 kg | 9.0000 mV | 9.0000 mV |
| Pressure unit converter | 1 bar → psi | 14.5037738 psi | 14.5037738 psi |
| Pressure unit converter | 1 atm → kPa | 101.325 kPa | 101.325 kPa |
| Pressure transmitter accuracy calculator | URL 100 · 0 … 40 bar · 0.075 % · 0.1 %/10 K × 20 K · 0.1 %/a × 1 a | 0.22561 bar | 0.22561 bar |
| Level to pressure calculator | 5 m · 998.21 kg/m³ · g 9.80665 m/s² | 0.489455 bar | 0.489455 bar |
| Dew point and humidity calculator | 20 °C · 50 % RH | 9.255 °C | 9.255 °C |
| Flow rate converter | 100 US gpm | 378.5411784 L/min | 378.5411784 L/min |
| Vibration converter | 2.8 mm/s RMS · 50 Hz | 0.126853 g pk | 0.126853 g pk |
| Strain gauge bridge calculator | quarter bridge · GF 2 · 1000 µε · 5 V | 2.4975 mV | 2.4975 mV |
| Strain gauge bridge calculator | full bridge · GF 2 · 1000 µε · 5 V | 10.0000 mV | 10.0000 mV |
| Temperature measurement error calculator | PT100 class A · 100 °C · 4-wire · 0 … 200 °C · 0.1 K · 0.03 % · 0.1 %/10 K × 20 K | 0.544151 K | 0.544151 K |
| Temperature measurement error calculator | type K class 1 · 500 °C · no other error | 2.0000 K | 2.0000 K |
| ADC resolution calculator | 16 bit · 0 … 2.5 V | 38.147 µV | 38.147 µV |
| Modbus register decoder | FLOAT32 · 0x4120 0x0000 · ABCD | 10.0000 | 10.0000 |
| Modbus register decoder | FLOAT32 · 0x0FDB 0x4049 · CDAB | 3.14159274 | 3.14159274 |
| Sensor accuracy spec converter | 0 … 100 · reading 50 · 0.1 % rdg + 0.05 % FS · 0.01 %/K × 10 K · 0.1 %/yr | 0.173205 | 0.173205 |
| NTC voltage divider calculator | 10 kΩ / β 3950 at 25 °C · 10 kΩ to 3.3 V · NTC to ground | -36.659 mV/K | -36.659 mV/K |
| Anti-aliasing RC filter calculator | 1 kΩ · 100 nF | 1591.549 Hz | 1591.549 Hz |
| Instrumentation amplifier gain calculator | K 49.4 kΩ · RG 499 Ω | 99.998 | 99.998 |
| Current transformer burden calculator | 5 A · 20 m of 2.5 mm² · 1 VA of devices | 7.88 VA | 7.88 VA |
| Sensor supply voltage drop calculator | 24 V · 50 mA · 200 m of 0.5 mm² | 23.312 V | 23.312 V |
| Pulse output calculator | 1024 PPR · ×4 · 3000 rpm | 204.8 kHz | 204.8 kHz |
| Battery life calculator | sleep 5 µA · 10 mA × 100 ms · 40 mA × 50 ms · every 60 s | 54.9875 µA | 54.9875 µA |
What this does not cover
- The calculators refuse input that is physically impossible, not input that is merely unusual. Within those limits they calculate exactly what you enter: the choice of values, and the engineering judgement behind them, stay with you.
- These checks cover the calculation, not your installation: sensor tolerance, wiring, instrument error and site conditions remain yours to assess.
- The Ex and IP decoders explain a marking. The certificate, the standard and the responsible person decide whether equipment may be used.
- Found a wrong number? Write to [email protected]. Corrections are checked against the standard and published.