Luminous by Greg Egan

Luminous by Greg Egan (Asimov’s SF, September 1995) gets off to a pretty good start with the story’s narrator waking to find himself handcuffed, and with a woman slicing into his biceps to get a data storage device. He tells her it is a “necrotrap,” and that the data will be destroyed if it is removed from his body—so she pauses and gets on the phone to order medical equipment which will fool the device. During this, he manages to spray her with his poisoned blood and, after she starts vomiting uncontrollably, eventually agrees to free him to get the antidote. As he departs he tells her there isn’t one, but that she’ll recover in twelve hours or so.
The next part of the story is something of a gear change, a flashback to a conversation between the narrator and a female student called Allison during a philosophy of mathematics course years earlier. This introduces the story’s gimmick, which is, if I’m not oversimplifying (note to self, write up Egan’s stories more promptly), that arithmetic may have different rules in other parts of the universe and this will fundamentally affect the nature of reality there.
The story then skips forward to the present, and we find the narrator is in China to meet Allison, who has arranged for them to have time on a supercomputer called Luminous to investigate this alternative arithmetic. We also get more math theory, and learn that (a) the other arithmetic system can theoretically be extended into our reality and (b) that this can be used for commercial advantage (there is a company called Industrial Algebra who are pursuing the pair to obtain their discovery so they can use it for financial gain—IA were behind the earlier biohacking attempt on the narrator).
The main part of the story occurs after they meet a Chinese professor and are taken to his laboratory:

Luminous was, literally, a computer made of light. It came into existence when a vacuum chamber, a cube five meters wide, was filled with an elaborate standing wave created by three vast arrays of high-powered lasers. A coherent electron beam was fed into the chamber—and just as a finely machined grating built of solid matter could diffract a beam of light, a sufficiently ordered (and sufficiently intense) configuration of light could diffract a beam of matter.
The electrons were redirected from layer to layer of the light cube, recombining and interfering at each stage, every change in their phase and intensity performing an appropriate computation—and the whole system could be reconfigured, nanosecond by nanosecond, into complex new “hardware” optimized for the calculations at hand. The auxiliary supercomputers controlling the laser arrays could design, and then instantly build, the perfect machine of light to carry out each particular stage of any program.

They then use their allotted half-hour to get the computer to map the extent of the “far side” and then discuss whether to contain it (so it can never be extended into out reality) or destroy it. Time constraints mean they chose the latter but (spoiler) the alternative mathematical reality begins to fight back . . .
There are several pages of back and forth in the struggle that ensues between the Luminous program and the alternative math, but they can’t contain the latter and eventually a spike of its reality reaches the lab they are in:

It hit me with a jolt of clarity more intense than anything I’d felt since childhood. It was like reliving the moment when the whole concept of numbers had finally snapped into place—but with an adult’s understanding of everything it opened up, everything it implied. It was a lightning-bolt revelation—but there was no taint of mystical confusion: no opiate haze of euphoria, no pseudo-sexual rush. In the clean-lined logic of the simplest concepts, I saw and understood exactly how the world worked—
—except that it was all wrong, it was all false, it was all impossible.
Quicksand.
Assailed by vertigo, I swept my gaze around the room—counting frantically: Six work stations. Two people. Six chairs. I grouped the work stations: three sets of two, two sets of three. One and five, two and four; four and two, five and one.
I weaved a dozen cross-checks for consistency—for sanity . . . but everything added up. They hadn’t stolen the old arithmetic; they’d merely blasted the new one into my head, on top of it.
Whoever had resisted our assault with Luminous had reached down with the spike and rewritten our neural metamathematics—the arithmetic that underlay our own reasoning about arithmetic—enough to let us glimpse what we’d been trying to destroy.

After this the narrator and Allison stop their attack, and the spike retreats. Then Allison realises that they can execute a program that will stop anyone using this other reality but which will not affect it. All ends well.
This is all a bit of a mixed bag—the good parts are very good (the opening, the computer battle with the alt maths, etc.) but unfortunately is stitched together with pages of dry math lectures and wild hand-wavium.
*** (Good). 13,000 words.