Issue 7

7. Timing / Synchronization Issues

This notebook documents the analysis for Issue #7: Timing / Synchronization Issues in the Autonity Oracle data. It covers:

What is this issue about?
Why conduct this issue analysis?
How to conduct this issue analysis?
What are the results?

7.1 What Is This Issue About?

Some validators submit their data earlier or later than others, creating synchronization or timing problems in the Oracle system. Possible symptoms include:

Large differences in timestamps among validators for the same minute or round.
Data consistently arriving late or early.
Potential clock skew or network delays.

This analysis examines how synchronized each validator’s submission timestamps are compared to the median submission timestamp within each minute.

7.2 Why Conduct This Issue Analysis?

Reliability: Timely, synchronized data submission is critical for accurate on-chain aggregation.
Diagnostics: Identifying validators with systematic timing offsets provides clear targets for correction before Mainnet launch.
Transparency: Documenting timing deviations clearly helps the team diagnose network or clock issues effectively.

7.3 How to Conduct the Analysis?

Use Python with the Polars library (v1.24.0) to:

Load and preprocess Oracle submission CSV files.
Parse timestamps from strings to actual datetimes.
Compute each validator’s submission offset (in seconds) relative to the median timestamp within each 30-second bin (re-anchored every 6 hours).
Summarize timing offsets per validator:
- Mean, median, max offsets
- Fraction of submissions exceeding thresholds (e.g., 30s, 60s)

Below is the script:

import polars as pl
import glob
import warnings

warnings.filterwarnings("ignore")

def load_and_preprocess_submissions(submission_glob: str) -> pl.DataFrame:
    """
    Loads Oracle Submission CSVs into a Polars DataFrame,
    parsing timestamps into datetime.
    """
    files = sorted(glob.glob(submission_glob))
    if not files:
        raise ValueError(f"No CSV files found matching pattern {submission_glob}")

    lf_list = []
    for f in files:
        lf_temp = pl.scan_csv(
            f,
            dtypes={"Timestamp": pl.Utf8},
            null_values=[""],
            ignore_errors=True,
        )
        lf_list.append(lf_temp)

    lf = pl.concat(lf_list)

    lf = lf.with_columns(
        pl.col("Timestamp")
        .str.strptime(pl.Datetime, strict=False)
        .alias("Timestamp_dt")
    )

    lf = lf.with_columns(
        [
            pl.col("Timestamp_dt").cast(pl.Date).alias("date_only"),
            pl.col("Timestamp_dt").dt.weekday().alias("weekday_num"),
        ]
    )

    df = lf.collect()

    return df


def compute_timing_offsets_30s_reanchor_6h(
    df: pl.DataFrame,
    chunk_hours: int = 6,
    period_seconds: int = 30
) -> pl.DataFrame:
    """
    Computes offsets by grouping submissions into ~30s bins,
    re-anchoring every 'chunk_hours' hours.
    """
    df_local = df.with_columns(
        (pl.col("Timestamp_dt").cast(pl.Int64) // 1_000_000_000).alias("epoch_seconds")
    )

    anchor_epoch = df_local.select(pl.min("epoch_seconds")).item()
    chunk_length_sec = chunk_hours * 3600

    df_local = df_local.with_columns(
        (
            (pl.col("epoch_seconds") - anchor_epoch) // chunk_length_sec
        ).alias("chunk_id")
    )

    df_local = df_local.with_columns(
        (
            pl.col("epoch_seconds")
            - (anchor_epoch + pl.col("chunk_id") * chunk_length_sec)
        ).alias("local_elapsed")
    )

    df_local = df_local.with_columns(
        (pl.col("local_elapsed") // period_seconds).alias("round_in_chunk")
    )

    df_local = df_local.with_columns(
        (
            pl.col("chunk_id").cast(pl.Utf8)
            + "-"
            + pl.col("round_in_chunk").cast(pl.Utf8)
        ).alias("round_label")
    )

    median_lf = (
        df_local.lazy()
        .group_by("round_label")
        .agg(pl.median("epoch_seconds").alias("median_epoch_seconds"))
    )

    df_with_median = (
        df_local.lazy()
        .join(median_lf, on="round_label", how="left")
        .with_columns(
            (pl.col("epoch_seconds") - pl.col("median_epoch_seconds"))
            .alias("offset_seconds")
        )
        .with_columns(
            pl.col("offset_seconds").abs().alias("abs_offset_seconds")
        )
    )

    return df_with_median.collect().sort(["Timestamp_dt", "Validator Address"])


def summarize_timing_offsets(df_offsets: pl.DataFrame) -> pl.DataFrame:
    """
    Summarizes computed offsets in timings per validator.
    """
    thresholds = [10, 30, 60, 300]

    def exceed_expr(t: int):
        return (
            (pl.col("abs_offset_seconds") > t)
            .cast(pl.Int64)
            .sum()
            .alias(f"exceed_{t}s_count")
        )

    agg_exprs = [
        pl.count("Validator Address").alias("total_submissions"),
        pl.mean("offset_seconds").alias("mean_offset_seconds"),
        pl.median("offset_seconds").alias("median_offset_seconds"),
        pl.max("abs_offset_seconds").alias("max_offset_seconds"),
    ] + [exceed_expr(t) for t in thresholds]

    lf_summary = (
        df_offsets.lazy()
        .group_by("Validator Address")
        .agg(agg_exprs)
        .with_columns(
            [
                (pl.col(f"exceed_{t}s_count") / pl.col("total_submissions"))
                .alias(f"fraction_exceed_{t}s")
                for t in thresholds
            ]
        )
    )
    return lf_summary.collect().sort("mean_offset_seconds")


def analyze_timing_synchronization_issues_30s_6h(
    submission_glob: str
) -> dict:
    """
    Main analysis function with 30s-based grouping and 6-hour re-anchoring.
    Returns a dict of DataFrames:
      - df_all_data:   The raw submission data
      - df_with_offsets:  The data with computed offsets
      - df_validator_offsets: Summaries per validator
    """
    df_all = load_and_preprocess_submissions(submission_glob)
    df_with_offsets = compute_timing_offsets_30s_reanchor_6h(df_all)
    df_validator_offsets = summarize_timing_offsets(df_with_offsets)

    return {
        "df_all_data": df_all,
        "df_with_offsets": df_with_offsets,
        "df_validator_offsets": df_validator_offsets,
    }

results = analyze_timing_synchronization_issues_30s_6h(
    submission_glob="../submission-data/Oracle_Submission_*.csv"
)

7.4 What are the results?

Below are summaries and interpretation based on the computed results.

Per-Submission Timing Offsets

# Preview submission offsets
results["df_with_offsets"]

shape: (171_957, 31)

Timestamp	Validator Address	AUD-USD Price	AUD-USD Confidence	CAD-USD Price	CAD-USD Confidence	EUR-USD Price	EUR-USD Confidence	GBP-USD Price	GBP-USD Confidence	JPY-USD Price	JPY-USD Confidence	SEK-USD Price	SEK-USD Confidence	ATN-USD Price	ATN-USD Confidence	NTN-USD Price	NTN-USD Confidence	NTN-ATN Price	NTN-ATN Confidence	Timestamp_dt	date_only	weekday_num	epoch_seconds	chunk_id	local_elapsed	round_in_chunk	round_label	median_epoch_seconds	offset_seconds	abs_offset_seconds
str	str	i64	i64	i64	i64	i64	i64	i64	i64	i64	i64	i64	i64	i64	i64	i64	i64	i64	i64	datetime[μs, UTC]	date	i8	i64	i64	i64	i64	str	f64	f64	f64
"2025-01-01T00:00:12+00:00"	"0x100E38f7BCEc53937BDd79ADE46F…	null	null	null	null	null	null	null	null	null	null	null	null	null	null	null	null	null	null	2025-01-01 00:00:12 UTC	2025-01-01	3	1735689	0	0	0	"0-0"	1.735704e6	-15.0	15.0
"2025-01-01T00:00:12+00:00"	"0x1476A65D7B5739dE1805d5130441…	620325937441701400	100	695598691900918500	100	1038118434882147900	100	1253219756960152400	100	6361566293421600	100	90498188759938100	100	719187673997958190	100	739468327258002540	100	1028199389385116100	100	2025-01-01 00:00:12 UTC	2025-01-01	3	1735689	0	0	0	"0-0"	1.735704e6	-15.0	15.0
"2025-01-01T00:00:12+00:00"	"0x197B2c44b887c4aC01243BDE7E4b…	621851874883402800	90	695797383801836900	90	1040636869764295700	90	1254849995231570000	90	6362132586843100	90	90668407500090700	90	719187673997958190	100	739468327258002540	100	1028199389385116100	100	2025-01-01 00:00:12 UTC	2025-01-01	3	1735689	0	0	0	"0-0"	1.735704e6	-15.0	15.0
"2025-01-01T00:00:12+00:00"	"0x1Be7f70BCf8393a7e4A5BcC66F6f…	621851874883402800	50	696815552923141200	50	1040636869764295700	50	1254849995231570000	50	6369426751592400	50	90668407500090700	50	719187695422512255	100	739470280548770193	100	1028202074723125300	100	2025-01-01 00:00:12 UTC	2025-01-01	3	1735689	0	0	0	"0-0"	1.735704e6	-15.0	15.0
"2025-01-01T00:00:12+00:00"	"0x22A76e194A49c9e5508Cd4A3E1cD…	618489241070407000	50	694800529715923900	50	1034998473377251800	50	1251589518688734700	50	6354941963946300	50	90327970019785400	50	719187673997958190	100	739468021811862294	100	1028198964675195000	100	2025-01-01 00:00:12 UTC	2025-01-01	3	1735689	0	0	0	"0-0"	1.735704e6	-15.0	15.0
…	…	…	…	…	…	…	…	…	…	…	…	…	…	…	…	…	…	…	…	…	…	…	…	…	…	…	…	…	…	…
"2025-01-01T23:59:44+00:00"	"0xdF239e0D5b4E6e820B0cFEF6972A…	null	null	null	null	null	null	null	null	null	null	null	null	null	null	null	null	null	null	2025-01-01 23:59:44 UTC	2025-01-01	3	1735775	0	86	2	"0-2"	1.735762e6	13.0	13.0
"2025-01-01T23:59:44+00:00"	"0xf10f56Bf0A28E0737c7e6bB0aF92…	618878902204408000	100	695394575677597300	100	1035639460950966400	100	1251439155469255300	100	6359816494648100	100	90434278037913900	100	722416634674059554	100	738461996489158416	100	1022210676007396000	100	2025-01-01 23:59:44 UTC	2025-01-01	3	1735775	0	86	2	"0-2"	1.735762e6	13.0	13.0
"2025-01-01T23:59:44+00:00"	"0xf34CD6c09a59d7D3d1a6C3dC231a…	619348846941496400	100	695394575677597300	100	1036552710966674300	100	1251905399877559600	100	6363149239103500	100	90507036482522300	100	722446500612394871	100	738492525770173755	100	1022210676007396000	100	2025-01-01 23:59:44 UTC	2025-01-01	3	1735775	0	86	2	"0-2"	1.735762e6	13.0	13.0
"2025-01-01T23:59:44+00:00"	"0xfD97FB8835d25740A2Da27c69762…	618878902204408000	100	695394575677597300	100	1035639460950966400	100	1251439155469255300	100	6359816494648100	100	90434278037913900	100	722446500612394871	100	738492525770173755	100	1022210676007396000	100	2025-01-01 23:59:44 UTC	2025-01-01	3	1735775	0	86	2	"0-2"	1.735762e6	13.0	13.0
"2025-01-01T23:59:45+00:00"	"0x19E356ebC20283fc74AF0BA4C179…	618878902204408000	100	695394575677597300	100	1035639460950966400	100	1251439155469255300	100	6359816494648100	100	90434278037913900	100	722446525464800496	100	738492525770173755	100	1022210640843001800	100	2025-01-01 23:59:45 UTC	2025-01-01	3	1735775	0	86	2	"0-2"	1.735762e6	13.0	13.0

The table above illustrates validators’ submission offsets compared to the median timestamp per minute:

Positive values indicate submissions later than median.
Negative values indicate earlier submissions.

Validator-Level Offset Summary

# Validator-level summary of timing offsets
results["df_validator_offsets"]

shape: (60, 13)

Validator Address	total_submissions	mean_offset_seconds	median_offset_seconds	max_offset_seconds	exceed_10s_count	exceed_30s_count	exceed_60s_count	exceed_300s_count	fraction_exceed_10s	fraction_exceed_30s	fraction_exceed_60s	fraction_exceed_300s
str	u32	f64	f64	f64	i64	i64	i64	i64	f64	f64	f64	f64
"0x1476A65D7B5739dE1805d5130441…	2462	-0.666531	-1.0	15.0	780	0	0	0	0.316816	0.0	0.0	0.0
"0x7232e75a8bFd8c9ab002BB3A00eA…	2877	-0.254432	0.0	15.0	778	0	0	0	0.270421	0.0	0.0	0.0
"0x831B837C3DA1B6c2AB68a690206b…	2863	-0.25358	0.0	15.0	777	0	0	0	0.271394	0.0	0.0	0.0
"0x6747c02DE7eb2099265e55715Ba2…	2840	-0.253169	0.0	15.0	770	0	0	0	0.271127	0.0	0.0	0.0
"0x01F788E4371a70D579C178Ea7F48…	2837	-0.251322	0.0	15.0	768	0	0	0	0.270708	0.0	0.0	0.0
…	…	…	…	…	…	…	…	…	…	…	…	…
"0x984A46Ec685Bb41A7BBb2bc39f80…	2877	-0.2374	0.0	15.0	780	0	0	0	0.271116	0.0	0.0	0.0
"0xd61a48b0e11B0Dc6b7Bd713B1012…	2873	-0.233554	0.0	15.0	779	0	0	0	0.271145	0.0	0.0	0.0
"0xbfDcAF35f52F9ef423ac8F2621F9…	2833	-0.232616	0.0	15.0	768	0	0	0	0.271091	0.0	0.0	0.0
"0x00a96aaED75015Bb44cED878D927…	2866	-0.223657	0.0	15.0	774	0	0	0	0.270063	0.0	0.0	0.0
"0x4cD134001EEF0843B9c69Ba9569d…	2823	-0.216082	0.0	15.0	766	0	0	0	0.271343	0.0	0.0	0.0

Interpretation of Validator Timing Offsets

Interpretation of validator-level offsets using the metrics above:

Mean Offset: Validators with high positive mean offsets (>20s) consistently submit late, suggesting potential clock or scheduling issues.
Median Offset: Confirms the consistency of early or late submissions.
Max Offset: Large values (>60s) suggest occasional severe delays or network disruptions.
Fraction Exceeding Thresholds: High fractions (>10%) indicate frequent timing deviations.

Validators with Significant Timing Issues

late_validators = results["df_validator_offsets"].filter(pl.col("mean_offset_seconds") > 30)
early_validators = results["df_validator_offsets"].filter(pl.col("mean_offset_seconds") < -30)

print("Consistently Late Validators (>30s delay):", late_validators["Validator Address"].to_list())
print("Consistently Early Validators (>30s early):", early_validators["Validator Address"].to_list())

Consistently Late Validators (>30s delay): []
Consistently Early Validators (>30s early): []

Validators listed as “Consistently Late” or “Early” warrant immediate investigation of clock synchronization or scheduling configurations.

Weekend vs. Weekday Offset Patterns

df_offsets = results["df_with_offsets"]
df_weekday_offset = df_offsets.group_by("weekday_num").agg(pl.mean("abs_offset_seconds").alias("avg_abs_offset")).sort("weekday_num")
df_weekday_offset

shape: (1, 2)

weekday_num	avg_abs_offset
i8	f64
3	7.219497

The above table reveals whether average absolute timing offsets differ substantially by day-of-week. Higher offsets on weekends could indicate decreased validator attention or configuration issues specific to weekends.

List of all Validators and their Mean of Abs Offset Seconds

df_offsets = results["df_with_offsets"]
df_mean_abs_offset = (
    df_offsets
    .group_by("Validator Address")
    .agg(
        pl.mean("abs_offset_seconds").alias("mean_abs_offset_seconds")
    )
    .sort("mean_abs_offset_seconds", descending=True)
)

for row in df_mean_abs_offset.to_dicts():
    print(
        f"Validator {row['Validator Address']}: "
        f"mean_abs_offset_seconds={row['mean_abs_offset_seconds']:.2f}"
    )

Validator 0x1476A65D7B5739dE1805d5130441A94022Ee49fe: mean_abs_offset_seconds=7.83
Validator 0x4cD134001EEF0843B9c69Ba9569d11fDcF4bd495: mean_abs_offset_seconds=7.22
Validator 0x2928FE5b911BCAf837cAd93eB9626E86a189f1dd: mean_abs_offset_seconds=7.22
Validator 0x6747c02DE7eb2099265e55715Ba2E03e8563D051: mean_abs_offset_seconds=7.22
Validator 0xd61a48b0e11B0Dc6b7Bd713B1012563c52591BAA: mean_abs_offset_seconds=7.21
Validator 0x831B837C3DA1B6c2AB68a690206bDfF368877E19: mean_abs_offset_seconds=7.21
Validator 0x01F788E4371a70D579C178Ea7F48E04e8B2CD743: mean_abs_offset_seconds=7.21
Validator 0x5E17e837DcBa2728C94f95c38fA8a47CB9C8818F: mean_abs_offset_seconds=7.21
Validator 0x984A46Ec685Bb41A7BBb2bc39f80C78410ff4057: mean_abs_offset_seconds=7.21
Validator 0x6a395dE946c0493157404E2b1947493c633f569E: mean_abs_offset_seconds=7.21
Validator 0x8f91e0ADF8065C3fFF92297267E02DF32C2978FF: mean_abs_offset_seconds=7.21
Validator 0x8584A78A9b94f332A34BBf24D2AF83367Da31894: mean_abs_offset_seconds=7.21
Validator 0xEf0Ba5e345C2C3937df5667A870Aae5105CAa3a5: mean_abs_offset_seconds=7.21
Validator 0xbfDcAF35f52F9ef423ac8F2621F9eef8be6dEd17: mean_abs_offset_seconds=7.21
Validator 0x3AaF7817618728ffEF81898E11A3171C33faAE41: mean_abs_offset_seconds=7.21
Validator 0xDF2D0052ea56A860443039619f6DAe4434bc0Ac4: mean_abs_offset_seconds=7.21
Validator 0xf34CD6c09a59d7D3d1a6C3dC231a7834E5615D6A: mean_abs_offset_seconds=7.21
Validator 0x59031767f20EA8F4a3d90d33aB0DAA2ca469Fd9a: mean_abs_offset_seconds=7.21
Validator 0xf10f56Bf0A28E0737c7e6bB0aF92f3DDad34aE6a: mean_abs_offset_seconds=7.21
Validator 0x94d28f08Ff81A80f4716C0a8EfC6CAC2Ec74d09E: mean_abs_offset_seconds=7.21
Validator 0xB5d8be2AB4b6d7E6be7Ea28E91b370223a06289f: mean_abs_offset_seconds=7.21
Validator 0x383A3c437d3F12f60E5fC990119468D3561EfBfc: mean_abs_offset_seconds=7.21
Validator 0x94470A842Ea4f44e668EB9C2AB81367b6Ce01772: mean_abs_offset_seconds=7.21
Validator 0x1Be7f70BCf8393a7e4A5BcC66F6f15d6e35cfBBC: mean_abs_offset_seconds=7.21
Validator 0x791A7F840ac11841cCB0FaA968B2e3a0Db930fCe: mean_abs_offset_seconds=7.21
Validator 0x3fe573552E14a0FC11Da25E43Fef11e16a785068: mean_abs_offset_seconds=7.21
Validator 0x23b4Be9536F93b8D550214912fD0e38417Ff7209: mean_abs_offset_seconds=7.21
Validator 0xDCA5DFF3D42f2db3C18dBE823380A0A81db49A7E: mean_abs_offset_seconds=7.21
Validator 0xE9FFF86CAdC3136b3D94948B8Fd23631EDaa2dE3: mean_abs_offset_seconds=7.21
Validator 0xfD97FB8835d25740A2Da27c69762D74F6A931858: mean_abs_offset_seconds=7.21
Validator 0x26E2724dBD14Fbd52be430B97043AA4c83F05852: mean_abs_offset_seconds=7.21
Validator 0xF9B38D02959379d43C764064dE201324d5e12931: mean_abs_offset_seconds=7.21
Validator 0x22A76e194A49c9e5508Cd4A3E1cD555D088ECB08: mean_abs_offset_seconds=7.21
Validator 0xE4686A4C6E63A8ab51B458c52EB779AEcf0B74f7: mean_abs_offset_seconds=7.21
Validator 0x551f3300FCFE0e392178b3542c009948008B2a9F: mean_abs_offset_seconds=7.21
Validator 0x718361fc3637199F24a2437331677D6B89a40519: mean_abs_offset_seconds=7.21
Validator 0x3597d2D42f8Fbbc82E8b1046048773aD6DDB717E: mean_abs_offset_seconds=7.21
Validator 0xcf716b3930d7cf6f2ADAD90A27c39fDc9D643BBd: mean_abs_offset_seconds=7.21
Validator 0xcdEed21b471b0Dc54faF74480A0E700fCc42a7b6: mean_abs_offset_seconds=7.21
Validator 0x358488a4EdCA493FCD87610dcd50c62c8A3Dd658: mean_abs_offset_seconds=7.21
Validator 0x9C7dAABb5101623340C925CFD6fF74088ff5672e: mean_abs_offset_seconds=7.21
Validator 0xBBf36374eb23968F25aecAEbb97BF3118f3c2fEC: mean_abs_offset_seconds=7.21
Validator 0x24915749B793375a8C93090AF19928aFF1CAEcb6: mean_abs_offset_seconds=7.21
Validator 0xC1F9acAF1824F6C906b35A0D2584D6E25077C7f5: mean_abs_offset_seconds=7.21
Validator 0x64F83c2538A646A550Ad9bEEb63427a377359DEE: mean_abs_offset_seconds=7.21
Validator 0x9d28e40E9Ec4789f9A0D17e421F76D8D0868EA44: mean_abs_offset_seconds=7.21
Validator 0x36142A4f36974e2935192A1111C39330aA296D3C: mean_abs_offset_seconds=7.21
Validator 0xD9fDab408dF7Ae751691BeC2efE3b713ba3f9C36: mean_abs_offset_seconds=7.21
Validator 0x5603caFE3313D0cf56Fd4bE4A2f606dD6E43F8Eb: mean_abs_offset_seconds=7.21
Validator 0xdF239e0D5b4E6e820B0cFEF6972A90893c2073AB: mean_abs_offset_seconds=7.21
Validator 0x197B2c44b887c4aC01243BDE7E4bBa8bd95BC3a8: mean_abs_offset_seconds=7.21
Validator 0x527192F3D2408C84087607b7feE1d0f907821E17: mean_abs_offset_seconds=7.21
Validator 0x99E2B4B27BDe92b42D04B6CF302cF564D2C13b74: mean_abs_offset_seconds=7.21
Validator 0x19E356ebC20283fc74AF0BA4C179502A1F62fA7B: mean_abs_offset_seconds=7.21
Validator 0x100E38f7BCEc53937BDd79ADE46F34362470577B: mean_abs_offset_seconds=7.21
Validator 0xBE287C82A786218E008FF97320b08244BE4A282c: mean_abs_offset_seconds=7.21
Validator 0xc5B9d978715F081E226cb28bADB7Ba4cde5f9775: mean_abs_offset_seconds=7.21
Validator 0xd625d50B0d087861c286d726eC51Cf4Bd9c54357: mean_abs_offset_seconds=7.21
Validator 0x7232e75a8bFd8c9ab002BB3A00eAa885BC72A6dd: mean_abs_offset_seconds=7.21
Validator 0x00a96aaED75015Bb44cED878D927dcb15ec1FF54: mean_abs_offset_seconds=7.20

Please note, a low mean_abs_offset_seconds indicates the validator typically submits very close to the group median, while a high value indicates they often drift too far from the typical submission time.